Remove duplicates

Shell Programming and Scripting
1

Hello Experts,

I have two files named old and new. Below are my example files. I need to compare and print the records that only exist in my new file. I tried the below awk script, this script works perfectly well if the records have exact match, the issue I have is my old file has got extra spaces/tabs at the end of the line and few blank lines at the beginning because of which the awk script produces the records that are in my old file, which I am not interested in.

I am looking for a solution that can strip the extra spaces/tabs at the end of the line
(or)
Instead of comparing the entire record I could compare just the 1st and 2nd field and if it matches with the 1st and 2nd field of my old file and produce the same result as shown. This might require a tweak in the awk script

nawk 'NR==FNR{a[$0];next}!($0 in a)' old new > sdiff

Old:
ACB_XY_01 1 hello
ACB_XY_03 1 hai
ACB_XY_04 1 good
ACB_XY_04 2 luck

New:
ACB_XY_01 1 hello
ACB_XY_01 2 hai
ACB_XY_03 1 hai
ACB_XY_04 1 good
ACB_XY_04 2 luck

Output:
ACB_XY_01 2 hai

Many thanks in advance.

2 
awk 'NR == FNR {
  _[$1,$2]
  next
  }
!(($1,$2) in _)
' Old New
3

Hello again,

Many thanks for your response. I tried the command with awk and nawk as below

awk 'NR == FNR {_[$1,$2] next }!(($1,$2) in _)' old new

but get an error message as

awk: syntax error near line1
awk: illegal statement near line1
nawk: syntax error at source line 1
context is
NR==FNR { _[$1,$2] >>> next <<< } ! (($1, $2) in _)
nawk: illegal statement at source line 1

Please can you tell me where i am going wrong

4

Yes,
in my post there is a newline before the next statement.
If you want to run it on one line, you should change the command:

awk 'NR == FNR { _[$1,$2]; next } !(($1,$2) in _)' old new
5

To use the command as a oneliner you have to place a command delimiter before the "next" statement

awk 'NR == FNR {_[$1,$2]; next }!(($1,$2) in _)' old new

Regards