Challenging Compare and validate question -- plus speed.

vgersh99 · May 25, 2006, 3:01pm

madhunk:

Yes....I tried to switch the order of the files in calling -- like this
nawk '
  FNR==NR{
   detail[$2]
   next
  }
  {
   if ( $2 in detail)
   printf("Metadata partner name [%s] found in Detail-- Successful\n",  $2)
   else {
   printf("Metadata partner name [%s] NOT found in Detail-- Failed\n",  $2)
   _ex=1
   }
  }
  END { exit(_ex)}' ${METADATA_FILE} ${DETAIL_FILE}
The problem is: I only have 5 records in Metadata file but I have 13 Million in the Detail file.

If $2 is there in the Detail file but not in the Metadata file, then I am getting this huge output of all the records.

Well it sounds like.... well.... everything is good - everything in META appears in Detail.

Are you sure you have any mismatches?
I'd suggest creating the a small sample 'mis-matched' pair of files and running the script.

madhunk · May 25, 2006, 3:38pm

Thank You again Vgersh99....

These are the sample files that I created..

Metadata File

M       ORBITZ     
M       AIRTRAN
M       FRONTIER
M       CAESAR
M       BESTWESTERN

Detail File

D       ORBITZ     
D       AIRTRAN
D       FRONTIER
D       MIDWEST
D       MIDWEST
D       MIDWEST
D       MIDWEST
D       BESTWESTERN

I am getting the output as

Detail partner name [ORBITZ] found in Metadata-- Successful
Detail partner name [AIRTRAN] found in Metadata-- Successful
Detail partner name [FRONTIER] found in Metadata-- Successful
Detail partner name [MIDWEST] NOT found in Metadata-- Failed
Detail partner name [MIDWEST] NOT found in Metadata-- Failed
Detail partner name [MIDWEST] NOT found in Metadata-- Failed
Detail partner name [MIDWEST] NOT found in Metadata-- Failed
Detail partner name [BESTWESTERN] found in Metadata-- Successful

If I have 5 million records for MIDWEST in the actual production file, the printf statement output will be huge.

I need the output printed out...But Is there any way that I could suppress the huge output and keep it to just one line like...Something like this

Detail partner name [ORBITZ] found in Metadata-- Successful
Detail partner name [AIRTRAN] found in Metadata-- Successful
Detail partner name [FRONTIER] found in Metadata-- Successful
Detail partner name [MIDWEST] NOT found in Metadata-- Failed
Detail partner name [BESTWESTERN] found in Metadata-- Successful

Please advice..

Thank You,
Madhu

vgersh99 · May 25, 2006, 4:54pm

look into 'man sort'

madhunk · May 25, 2006, 5:48pm

I did vgersh99...

I understand we can use

sort -u

to output only unique values...But not sure how I can use in the current code to suppress the output to only one row..

vgersh99 · May 25, 2006, 6:46pm

You should really start experimenting yourself and stop relying on the complete solutions on these forums.

nawk '.....' File1 File2 | sort -u

madhunk · May 26, 2006, 9:55am

I apologize vgersh99...I was trying my best and I am new to this whole scripting world....

Thank you and will experiment a little more before I ask the forum..

Have a nice day!