Some Awk Getline help?

Greetings,

I have about 3000 files that I want to search. The first column in all of these 3000 files has a unique serial number on each line. The subsequent columns have lots of data.
I have another masterfile with three columns to help me find all the data I need in a moments notice:

col 1 col2 col3
serial.1 row# file1
serial.2 row# file345
serial.3 row# file1023

what I want to do take list of serial numbers, get the file name and row number for where the data sits, and then spit out the data into one file. I figured awk getline might help call up the filename and row name to go searching, but I am really confused with this command.
Another thought was to make a shell script to go down each row in the masterfile, create a variable for "row number variable" and one for "filenamevariable" and then awk '(NF=rownumbervariable){print $0}' filenamevariable >>outputfile

but cant get this working either.

ANy suggestions?

Thanks,
jeeplou

Welcome to the Forum,

Can you please post the sample input and expected output ( might be you can place two or three files ).

1. large datafiles to be queried called "file.dat*" (column one is serial number, others are data)

file.dat1

rs10001 900 900 100 100
rs10002 800 300 200 100

file.dat2
rs10003 222 111 333 444
rs10004 999 121 232 434

2. small masterlist/cheatsheet called "master.list" with three columns: serial#,row#,filename

rs10001 1 file.dat1
rs10002 2 file.dat1
rs10003 1 file.dat2
rs10004 2 file.dat2

3. If I want to recover rs10001 and rs10004 data, I grep from my masterlist to create a subset of insterest called subset.txt

subset.txt
rs10001 1 file.dat1
rs10004 2 file.dat2

Now, I need to recover the original data for each serial number with the expected output:

rs10001 900 900 100 100
rs10004 999 121 232 434

Thanks,
jeeplou

Try this:

awk 'NR==FNR{a[$1]=$1;next}a[$1]' subset.txt file.dat*

thanks for the speedy reply. I understand each individual element in your code, but since I'm not sure what it is "saying", I'm not sure how to set this up for my data....can you elaborate briefly please?

also, is it not helpful to use the file name and row number to help awk figure out where to look?

Have you tried the command?

Place file.dat1, file.dat2 and subset.txt in a directory, run the command to see what happens.

it most definitely works!
what I understand from the code is that you are creating an array that puts all of the serial numbers together and then testing each line of the array as a pattern in all of the files.

My concern is the length of time this will take over 64GB of data...I havent tried this yet, but will get the LSF farm going to see what happens. Is there a way to use the file name and row number to speed things up?

thanks again.
jonah

Perhaps you are looking for something a bit like this? This would probably limit the searching:

awk '$3!=f{close(f);f=$3;n=0}{while(!p && getline x<f) if(++n==$2){p=1};print x;p=0}' subset.txt

This will open and close only those files that are specified in subset.txt and look for the right line number and print the record it finds on that line. For efficiency reasons subset.txt would best be sorted..

Output:

rs10001 900 900 100 100
rs10004 999 121 232 434

Create a file with the desired serial# with the name file:

$ cat file
rs10001
rs10004

Now you can run this command in the directory with the files:

awk  '
NR==FNR{
  a[$1]=$1
  next
}
a[$1] {row[$1]=$2; file[$1]=$3}
END{
  for(i in file){
    c=0
    while((getline < file) > 0) {
      ++c
      if(c==row) {
        print
      }
    }
  }
}' file masterfile

hmm..it's interesting but it is bringing up a syntax error but doesnt say exactly... I cut and paste directly. Any thoughts?

Thanks

---------- Post updated at 03:56 PM ---------- Previous update was at 03:34 PM ----------

It doesn't like the while (!p &&.....)
it keeps saying that p: event not found

That is a bash message. It typically means that quoting is not done right and bash interprets the "!" . Did you use the single quotes?

I tried single and double quotes around the entire awk statement and they both returned the same error. Should I put the quotes elsewhere?

here is the code I tried:

awk '$3!=f{close(f);f=$3;n=0}{while(!p && getline x<f ) if(++n==$2){p=1};print x; p=0   }' testfile

Incidentally, what intuitively is meant by while (!p && getline x<f)

I understand the second part to mean "as long getline returns an actual line from file 'f' "
but I'm not understanding where the 'p' comes in. I see the downstream code for 'p' but it still isn't sinking in.

Strange. Are you on Solaris? If so use nawk or /usr/xpg4/bin/awk instead of standard awk.

It means while the variable p is empty or zero AND reading a line was successful (in other words no EOF was encountered). If the variable p equals 0 then !p equals 1 (i.e. "TRUE")

it worked great and fast! But instead of !p, I used p==0. Do you see any nasty side effects of doing that workaround?

No, no side effects. That should work too.

much appreciated. thanks again.