Need some Help for file filteration and saving the output in other directory

ektubbe · February 19, 2012, 7:48am

Hi all........
Plss do help me.......in a big trouble... :wall::wall::wall:
I have 3 directories named as :1. /home/shuchi/source
2./home/shuchi/destination
3./home/shuchi/filter
now the problem is /home/shuchi/source has say 2 files with extension .txt as given below :
A.txt

msisdn             | imsi | name | subscription | product | date | recharge type
09311424354 | 09488439820 | ekta | yes | VAS | 2012-12-07 | IVR
09128734473 | 83974738734 | priya | no | Non VAS | 2011-08-11 | SMS
09376536263 | 63473627632 | shubhra | yes | VAS | 2011-07-07 | WAP
09655423434 | 13312353343 | shubhi | no | non VAS | 2011-05-04 | SMS
09473874383 | 36267267372 | amita | yes | VAS | 2011-04-05 | IVR

B.txt

msisdn             | imsi | name | subscription | product | date | recharge type
09311424354 | 09488439820 | ekta | yes | VAS | 2012-12-07 | IVR
09128734473 | 83974738734 | priya | no | Non VAS | 2011-08-11 | SMS
09376536263 | 63473627632 | shubhra | yes | VAS | 2011-07-07 | WAP
09302117719 | 13312353343 | shubhi | no | non VAS | 2011-05-04 | SMS
08295708671 | 36267267372 | amita | yes | VAS | 2011-04-05 | IVR

/home/shuchi/filter has one file as below :
msisdn
--------------

09311424354
09128734473
09487736472
09302117719
08295708671

now on the basis of file present in /home/shuchi/filter i need to have 2 subfiles with the same name as A.txt and B.txt in /home/dstst/destination folder as given below :
A.txt

msisdn             | imsi | name | subscription | product | date | recharge type
09311424354 | 09488439820 | ekta | yes | VAS | 2012-12-07 | IVR
09128734473 | 83974738734 | priya | no | Non VAS | 2011-08-11 | SMS:o

B.txt

msisdn             | imsi | name | subscription | product | date | recharge type
09302117719 | 13312353343 | shubhi | no | non VAS | 2011-05-04 | SMS
08295708671 | 36267267372 | amita | yes | VAS | 2011-04-05 | IVR

i have to write a script which will read files present in /home/shuchi/source one by one i.e. A.txt and B.txt and match them with file present in /home/shuchi/filter and on the basis of msisdn and store the result in /home/shuchi/destination directory.I have to use grep or fgrep as some crappy version of unix being used and awk and sed are not supported ............it would be a grttttttttt help...........:(

~Priya...........

agama · February 19, 2012, 9:13am

No awk, that's very odd. Assuming you have Kshell or bash, have a go with this. Replace the directory names with yours (I used the ones here to test).

#!/usr/bin/env bash
# change the directory names to be your source, destination and filter
sdir=src     
ddir=dest
fdir=filter

# use find to prevent issues if there are a lot of *.txt files
find $sdir -name "*.txt" | while read file
do
    grep -f $fdir/filter $file >$ddir/${file##*/}
done

You might want to remove the line of dashes from your filter file since that is not a potential pattern in one of your source files you'll waste time with each line in each grep trying to match it.

ektubbe · February 21, 2012, 7:21am

agama:

No awk, that's very odd. Assuming you have Kshell or bash, have a go with this. Replace the directory names with yours (I used the ones here to test).
#!/usr/bin/env bash
# change the directory names to be your source, destination and filter
sdir=src     
ddir=dest
fdir=filter

# use find to prevent issues if there are a lot of *.txt files
find $sdir -name "*.txt" | while read file
do
   grep -f $fdir/filter $file >$ddir/${file##*/}
done
You might want to remove the line of dashes from your filter file since that is not a potential pattern in one of your source files you'll waste time with each line in each grep trying to match it.

Thanks a lot buddy....it worked....thx again....

---------- Post updated at 07:21 AM ---------- Previous update was at 03:38 AM ----------

agama:

No awk, that's very odd. Assuming you have Kshell or bash, have a go with this. Replace the directory names with yours (I used the ones here to test).
#!/usr/bin/env bash
# change the directory names to be your source, destination and filter
sdir=src     
ddir=dest
fdir=filter

# use find to prevent issues if there are a lot of *.txt files
find $sdir -name "*.txt" | while read file
do
   grep -f $fdir/filter $file >$ddir/${file##*/}
done
You might want to remove the line of dashes from your filter file since that is not a potential pattern in one of your source files you'll waste time with each line in each grep trying to match it.

frnd...........:wall::wall::wall:
d above script is writing the file to destination directory but no data in it...
all d files are 0 byte files....my manager is makin my lyf hell to complete dis...
plz plz do help me.................:wall:

~Priya............

agama · February 21, 2012, 5:20pm

If you could paste the script you are using it would help. Possibly a typo in the script. You could also try running a grep by hand to see what it yields.

Assuming one of the files that find will list is foo.txt, try running this to see what happens:

grep -f filter-file-name foo.txt|more

If you see output from this command, then your script likely has a bug. No output could mean that there was nothing in foo.txt to find, or the filter file is wrong.

Hard to say more without seeing your script.

ektubbe · February 22, 2012, 12:25am

agama:

If you could paste the script you are using it would help. Possibly a typo in the script. You could also try running a grep by hand to see what it yields.

Assuming one of the files that find will list is foo.txt, try running this to see what happens:
grep -f filter-file-name foo.txt|more
If you see output from this command, then your script likely has a bug. No output could mean that there was nothing in foo.txt to find, or the filter file is wrong.

Hard to say more without seeing your script.

#!/usr/bin/env bash
# change the directory names to be your source, destination and filter
sdir=/data1/T2EDW/R1D1_SIT/initial_files/00800010
ddir=/home/dstst/ekta
fdir=/home/dstst/ekta/xp

# use find to prevent issues if there are a lot of *.txt files
find $sdir -name "*.EDW" | while read file
do
    grep -f $fdir $file >> $ddir/${file##*/}
done

above is the script with proper directories for source and destinations...plz plz help me......my manager is at top of my head plz............:wall::wall::(

agama · February 22, 2012, 6:09pm

I don't see any problems with your script. You probably should change the double redirection (>>) to a signle (>), but that's not causing the files to be empty.

The only thing I can think of is that you might have trailing blanks and/or tabs in your list of patterns (the filter file: /home/dstst/ekta/xp). Check it to be sure that there isn't any leading or trailing white space (spaces or tabs) as that might be the cause of nothing being found.