Extracting unique values of a column from a feed file

Shell Programming and Scripting
1

Hi Folks,

I have the below feed file named abc1.txt in which you can see there is a title and below is the respective values in the rows and it is completely pipe delimited file ,.

ABC_ID|AMOUNT|ABC_DATE|ABC_CODE|ABD_ID|ABDE_ID|ABEFF_DATE|ABAL_AMOUNT|ab_ON|AB_ODE|TY_CODE|RITY_TE|CR_OKER|SYS_FLAG|CRT_MENT|ADM_ID|ERG_ID|ASH_ADE
0|0.00|24-Jun-14|SSRD|1677|82588|20-Mar-14|100004.00|0|Serest|TRRS|24-Mar-19||true|Receive|861|0|3862880
1|0.00|24-Sep-14|SRSD|1477|85288|20-Mar-14|100003.00|0|Serest|TYRS|24-Mar-19||true|Receive|831|0|3828680
2|0.00|24-Dec-14|HHSD|1777|82858|20-Mar-14|100006.00|0|Serest|UIRS|24-Mar-19||true|Receive|811|0|3862880
2|0.00|24-Dec-14|ESJD|1877|82885|20-Mar-14|100009.00|0|Serest|OPRS|24-Mar-19||true|Receive|861|0|3682880

now this feed files is been generated regularly by a process and is being kept at unix box at the following location /usr/cft/str so finally the file is at /usr/cft/str/abc1.txt
now from the right side you can see there is a column named ADM_ID ,
can you please advise the script or command that will extract the unique ADM_ID and will store those unique ADM_ID in a newly created file the name of the newly created file is Unique_ADM and this file will be stored at the same location .

so the newly created Unique_ADM file will contain the following data that it will extract from the above feed file..

861
831
811

please advise how to achieve this.:eek:

2

You could try something like:

#!/bin/ksh
cd /usr/cft/str/  || exit 1
awk -F'|' -v id='ADM_ID' '
NR == 1 {
        for(f = 1; f <= NF; f++)
                if($f == id)
                        break
        if(f > NF) {
                printf("Column header \"%s\" not found.\n" id)
                exit 2
        }
        next
}
x[$f]++ == 0 {
        print $f
}' abc1.txt > Unique_ADM

This was tested using the Korn shell, but it will work with any shell that recognizes basic Bourne shell syntax.

3

Specifically for ADM_ID, not flexible as Don's code

awk -F'|' 'NR>1 && $0=$16' infile | sort -u
4

@ahamed, there would be an issue if $16=0 , also there was an issue with BSD awk which did not print everything.. Rearranging your script like this, worked though:

awk -F'|' '{$0=$16}NR>1' infile | sort -u
1 Like
5 
sed '1 d' fo|cut -'d|' -f16|sort|uniq
811
831
861

easiest way to do is,in case if you want an explanations whats happening

sed 1d is selecting all but first line and rest all is self explanatory