Help with merge and remove duplicates

Shell Programming and Scripting
1

Hi all,

I need some help to remove duplicates from a file before merging.

I have got 2 files:

file1 has data in format
4300 23456
4301 2357

the 4 byte values on the right hand side is uniq, and are not repeated anywhere in the file

file 2 has data in same format but is not in sorted order

3999 2349
4305 1234
4300 3459

the script i am using is not able to remove duplicate values after spaces, and the output is coming in the following manner:-

3999 2349
4300 233445569
4301 2357
4305 1234

The expected output is:-
3999 2349
4300 234569
4301 2357
4305 1234

The script i am using is:-

export INPUT1=File1
export REQ=File2
export OUTPUT1=Final_File


sort $INPUT1 $REQ |
           awk     '$1 == SV1              {SV2=SV2  $2;next}
           NR>1 && $1 != SV1                {print SV1, SV2}
                                            {SV1=$1;SV2=$2}
           END                             {print  SV1,SV2}
        ' > $OUTPUT1
2

I don't see how the 'expected' is different from 'produced'...
Could you provide a better (more representative) set of sample files to explain what you're after.....

3

For the value 4300

produced value is:
4300 233445569

expected value is:-
4300 234569

4

You mean you want to change 233445569 to this 234569? and not the records but the repeated values in the fields?

6

yes....thats correct....

7

We need to see your script. How is the value from first file 4300 23456 and second file 4300 3459 ending up as 4300 233445569 ?

8

The script is given in the 1st post.......................

9

hmmm. I don't quite follow how you get from one to the other given 2 files....
Is there a common field between the 2 files?
Please explain verbally - never mind the script....

10

there are 2 files, both have the value 4300 followed by some codes.
Now we get a request to update the codes.

So the 1st file is the original file which has the value 4300 followed by some codes. The 2nd file also has the value 4300 followed by some more codes which are to be appended.

so when i append the files, i am not able to remove the duplocates after the value 4300 so the result is coming as:-
4300 233445569

Suppose 4300 is a key against which we need to update the values........

11

please define 'append the field'.
Are concatenating 2 fields in some manner (if so how) and then want to remove sequential digits/letters of the same value?
This is still too vague of a requirement. Please take one record from both files (say 4300) and show how you're 'appending' and how you're 'removing duplicates'.

12

for gnu awk, try:

awk '
   NR == FNR { a[$1]=$2 ; b[$1]=$0 ; next}
   {b[$1]= $1 " " $2 a[$1]}
   END {
      for (i in b) {
         fc=b; sub(" .*", "", fc);
         bts=b; sub(".* ", "", bts);
         for (n=1; n<=length(bts); n++) ba[substr(bts,n,1)]=substr(bts,n,1);
         ic=asort(ba,ab) ; bts="";
         for (j=1; j<=ic; j++) {bts=bts ab[j]; delete ba[ab[j]]; delete ab[j]; }
         print fc, bts;
      }
   }
' File1 File2 | sort