Uniq based on first field

Hi New to unix.
I want to display only the unrepeated lines from a file using first field.

Ex:

1234 uname1 status1
1235 uname2 status2
1234 uname3 status3
1236 uname5 status5

I used

sort filename | uniq -u 

output:

1234 uname1 status1
1235 uname2 status2
1234 uname3 status3
1236 uname5 status5

but here uniqness is happening based on all fields.

I want the output like this.

1235 uname2 status2
1236 uname5 status5

want to remove repeated lines using first field only.

I tried a lot but unable to find solution. Anybody please help me out...

 
$ for i in `nawk '{print $1}' test`; do count=`grep -c $i test`; [ "$count" -gt "1" ] || grep $i test; done                                        
1235 uname2 status2
1236 uname5 status5

$ cat test
1234 uname1 status1
1235 uname2 status2
1234 uname3 status3
1236 uname5 status5

1 Like
awk 'NR==FNR{a[$1]++; next} a[$1]==1 ' infile infile
1 Like

Thanks to itkamaraj and rdcwayx for prompt reply.

@rdcwayx
Could you please explain the code how it works?
and If my file have , as separator[.csv file] then where I have to change the code?

ex:
1234,uname1,status1
1235,uname2,status2
1234,uname3,status3
1236,uname5,status5

Thanks,
Venu.

Use -F option:

awk -F',' 'NR==FNR{a[$1]++; next} a[$1]==1 ' infile infile
1 Like

If your first field is always 4 bytes and you have GNU uniq:

sort filename | uniq -u -w4

@rdcwayx
The above code is working fine. But, when my input file reached 11 millions then the code is producing empty file. Could you please help me out from this issue?

Thanks,
venu.

why can you not use sort?

sort -u -t ',' -k1.1,1.5 infile > outfile

@itkamaraj
I used your code also, but got 'no space' error. I think these codes are doing manipulation on memory. could you please explain your code ?

if your file has 11 million lines, then go for perl coding

Thanks for the reply. I want to display only the unrepeated lines.
I think -u will give us single line if it founds duplicates.
and here 1.1 to 1.5 means the length of first filed. isn't it?