Script to compare 1 file with all others

Shell Programming and Scripting
1

I need a script to compare 1 file with all others then print off any new lines which the other files do not have.

Running:

Solaris 10 5/09 s10s_u7wos_08 SPARC

Any help or pointers would be appreciated

2

Try:

for i in *; do comm -23 <(sort file_1) <(sort $i); done

Didn't test it.

3

can you explain what each bit does? as I'm not quite there yet

4

First tell if it worked for you :smiley:

5

I used

for i in *; do comm -23 <(sort 20111222.csv) <(sort $*csv); done

but it seemed to print everything off?

I need it to compare 20111222.csv with *csv then print off only new lines..

:slight_smile:

---------- Post updated at 12:12 AM ---------- Previous update was at 12:02 AM ----------

say the files typically have lines like so:

NE:220036,SHELF:1,SLOT:4,19:30:02,Thu Dec 22 2011
NE:223334,SHELF:9,SLOT:5,19:30:02,Thu Dec 22 2011
NE:877351,SHELF:2,SLOT:6,21:00:01,Thu Dec 22 2011
NE:222544,SHELF:2,SLOT:1,22:30:01,Thu Dec 22 2011

I need it to compare only NE:222544,SHELF:2,SLOT:1 from one file against all others to see if there is a match or not, if there isn't print it off as a new line, the ,22:30:01,Thu Dec 22 2011 bit i want it to not compare but ignore

sorry

6

Try this...

awk -F, 'NR==FNR{a[$1$2$3]++;next} !a[$1$2$3]' 20111222.csv *.csv

--ahamed

7

tried that this is the output

#awk -F, 'NR==FNR{a[$1$2$3]++;next} !a[$1$2$3]' 20111223.csv *.csv
awk: syntax error near line 1
awk: bailing out near line 1
8

Which is your OS? If solaris, use nawk

--ahamed

9

Solaris 10

---------- Post updated at 09:01 PM ---------- Previous update was at 08:51 PM ----------

OK tried that got output but not really "1st time offenders" lots of other stuff

can you advise what each part of this does?

nawk -F, 'NR==FNR{a[$1$2$3]++;next} !a[$1$2$3]' 20111223.csv *.csv

---------- Post updated at 09:08 PM ---------- Previous update was at 09:01 PM ----------

I need the file 20111223.csv to compare against *.csv and look for NE:506723,SHELF:11,SLOT:4 if a match is found tally up all the same results, if no match is found and it is a new line then highlight with "NEW" and tally up i.e if it's repeated many times in the new file 20111223.csv

The script can ignore the following lines ,06:30:04,Wed Dec 21 2011

typical line in file is:

NE:506723,SHELF:11,SLOT:4,06:30:04,Wed Dec 21 2011
10 
root@bt:/tmp# cat 20111222.csv
NE:222544,SHELF:2,SLOT:1,22:30:01,Thu Dec 22 2011

root@bt:/tmp# cat 1.csv
NE:220036,SHELF:1,SLOT:4,19:30:02,Thu Dec 22 2011
NE:223334,SHELF:9,SLOT:5,19:30:02,Thu Dec 22 2011
NE:877351,SHELF:2,SLOT:6,21:00:01,Thu Dec 22 2011
NE:222544,SHELF:2,SLOT:1,22:30:01,Thu Dec 22 2011

root@bt:/tmp# awk -F, 'NR==FNR{a[$1$2$3]++;next} !a[$1$2$3]' 20111222.csv *.csv
NE:220036,SHELF:1,SLOT:4,19:30:02,Thu Dec 22 2011
NE:223334,SHELF:9,SLOT:5,19:30:02,Thu Dec 22 2011
NE:877351,SHELF:2,SLOT:6,21:00:01,Thu Dec 22 2011

Isn't this what you are looking for? Let me know...

--ahamed

11

not quite, i need it to not have the date information only match NE:220036,SHELF:1,SLOT:4 the rest it can ignore - can you understand my requirement from above?

appreciate your help on this.

:wall:

12

The date is not being compared here, it is only being displayed. Only first 3 fields are checked i.e. $1 $2 and $3

You can try it out. If you don't want the date to be displayed, we can do that also.

nawk -F, 'NR==FNR{a[$1$2$3]++;next} !a[$1$2$3]{print "NEW: "$1,$2,$3}' OFS="," 20111222.csv *.csv

--ahamed

13

brilliant.

can you add to that a number of previously matched lines i.e 33

14

Didn't quite get you... 33?

--ahamed

15

ok here's my problem your script compares say two files fine but when there are multiple files it prints of wierd stuff for example:

#cat 20111224.csv
NE:852068,SHELF:1,SLOT:5,00:00:01,Sat Dec 24 2011
NE:564127,SHELF:9,SLOT:3,00:30:02,Sat Dec 24 2011
NE:225827,SHELF:8,SLOT:5,01:00:02,Sat Dec 24 2011

#nawk -F, 'NR==FNR{a[$1$2$3]++;next} !a[$1$2$3]{print "NEW: "$1,$2,$3}' OFS="," 20111224.csv *.csv
NEW: NE:851528,SHELF:3,SLOT:5
NEW: NE:598771,SHELF:9,SLOT:4
NEW: NE:828273,SHELF:2,SLOT:3
NEW: NE:219865,SHELF:4,SLOT:6
NEW: NE:237642,SHELF:12,SLOT:3
NEW: NE:848649,SHELF:4,SLOT:2
NEW: NE:851608,SHELF:5,SLOT:5
NEW: NE:219961,SHELF:5,SLOT:2
NEW: NE:877072,SHELF:2,SLOT:5
NEW: NE:222544,SHELF:15,SLOT:3
NEW: NE:560725,SHELF:1,SLOT:4
NEW: NE:560725,SHELF:1,SLOT:4
NEW: NE:851528,SHELF:3,SLOT:4
NEW: NE:858436,SHELF:4,SLOT:1
NEW: NE:221809,SHELF:12,SLOT:6
NEW: NE:220172,SHELF:1,SLOT:5
NEW: NE:560725,SHELF:13,SLOT:6
NEW: NE:565538,SHELF:6,SLOT:1
NEW: NE:220172,SHELF:1,SLOT:5
NEW: NE:594728,SHELF:2,SLOT:2
NEW: NE:505303,SHELF:3,SLOT:1
NEW: NE:571456,SHELF:5,SLOT:5
NEW: NE:560725,SHELF:16,SLOT:4
NEW: NE:220172,SHELF:1,SLOT:5
NEW: NE:860788,SHELF:2,SLOT:6

The list just goes on and on...

by 33 i mean

NE:860788,SHELF:2,SLOT:6                        33
NE:828273,SHELF:2,SLOT:3                        84

i.e matches the same text then increments the counter for that match.

really appreciate the help on this.

16

Try this...

awk -F, 'NR==FNR{a[$1$2$3]++;next} !a[$1$2$3]{b[$1OFS$2OFS$3]++}END{for(i in b){print "NEW: "i"\t"b}}' OFS="," 20111222.csv *.csv

--ahamed

17

Great ahamed were getting there, so close - you are a genius !

but two problems now

  1. I need only the counter for file 20111224.csv in the output & not every other .csv file (but want it to compare every other .csv file if that makes sense?)
  2. I don't see any "NEW" lines with that last script

In summary

1.It needs to be a summary of 20111224.csv file comparing against all other records (with a numeric value at the end of each line for only 20111224.csv file)
2.If no match is found for a line in the 20111224.csv file comparing against all other files then the word "NEW" appears for that line

18

No, your #1 does not make sense. Because the output will not have anything from 20111224.csv, only lines not present in 20111224.csv are displayed and that is what you have asked for.

#2 - Check the previous post, I have updated it.

--ahamed

---------- Post updated at 07:59 PM ---------- Previous update was at 07:55 PM ----------

Try this..

awk -F, 'NR==FNR{a[$1OFS$2OFS$3]++;next}!a[$1OFS$2OFS$3]{b[$1OFS$2OFS$3]++}
END{for(i in b){print "NEW: "i}for(i in a){if(a>0)print i"\t"a}}' OFS="," 20111222.csv *.csv

Above code will eliminate the duplicates in other files. If you want to retain the duplicate, try this...

awk -F, 'NR==FNR{a[$1OFS$2OFS$3]++;next}!a[$1OFS$2OFS$3]{print "NEW: "$1,$2,$3}
END{for(i in a){if(a>0)print i"\t"a}}' OFS="," 20111222.csv *.csv

--ahamed

19

ok can I flip this around and ask that the 20111224.csv file (which is the latest data) has it's lines with numerics against them and "NEW" for no matches against all other files

20

Check the previous post, it should give you want you want.

--ahamed