Need Help with a Data Extract script

hi,

i'am trying to write a shell program which takes a data element
from one file(var/tmp/usr1/pools)and search that data element
in a zipped file in archive location/usr01/archive/PSta*.Z).

cat var/tmp/usr1/pools
FC5173
FI5178
BE5221
FE5229
ST1604

ls /usr01/archive/PSta*.Z
/usr01/archive/PStat.dat.021711.001609.Z
/usr01/archive/PStat.dat.021811.001608.Z
/usr01/archive/PStat.dat.021911.001608.Z
/usr01/archive/PStat.dat.022311.001608.Z
/usr01/archive/PStat.dat.022411.001608.Z
/usr01/archive/PStat.dat.022511.001608.Z
/usr01/archive/PStat.dat.022611.001608.Z
/usr01/archive/PStat.dat.022811.190155.Z

below program writes that zipped filename into a temp file(/var/tmp/usr1/fcount) if it finds the occurence.

#!/usr/bin/sh
fl='/usr01/archive/'
for i in `cat /var/tmp/usr1/pools`
do
for j in `ls ${fl}/*.Z | tr -s ' '| cut -d" " -f9 | cut -d\/ -f7`
do
count=`zcat $fl/$j | grep "$i" | wc -l` 
if [ $count -eq 1 ] ; then
echo ${j} | grep "PSta*" >> /var/tmp/usr1/fcount
fi
done
done
 

lets consider that the data element found two files having it's occurence and wrote
two filenames to the temp file, Now my goal is to find the latest filename having that
occurence(data element).
the program is getting all the file names into /var/tmp/usr1/fcount where i 'am not able to get the latest
file for each data element
i've been trying without any success..will appreciate if any one help me out here.
thanks
kmr023

That's a useless use of cat and useless use of backticks, inefficient and potentially dangerous since there's a limit to how much data can be put into one variable.

That's also a useless use of ls *. Just do for j in ${fl}/* .

'for j in *' has the same size limit as the backticks problem. If you have more than a dozen or two files, or the number of files continually grows, this may eventually become a problem.

I don't quite understand the purpose of for j in `ls ${fl}/*.Z | tr -s ' '| cut -d" " -f9 | cut -d\/ -f7` . Are you rewriting your own basename? I don't see any purpose for basename here either.

I don't think grep "PSta*" does what you think it does. grep uses regular expressions, not shell globs, so "a*" here means "zero or more 'a' characters". So it'd match any line containing PSt at all. Just the expression 'PSta' should suffice if you want to find any line with those four letters.

You don't need to pipe grep into wc -l to measure whether grep found any results. grep's return code tells you whether it found anything, which you can feed directly into an ordinary if statement:

if grep pattern file > /dev/null
then
        echo 'Found'
else
        echo 'not found'
fi

grep can do nearly your entire program by itself, using -f and -F to tell it to read patterns from files instead of the commandline.

My version:

for FILE in /usr01/archive/PStat*.Z
do
        # Pipe the decompressed file into grep.  Use a pattern file (-f var/tmp/usr1/pools),
        # match fixed strings instead of regexes (-F), and stop on the
        # first match(-m 1) so scanning the entire file isn't always necessary.
        if zcat "$FILE" | grep -m 1 -F -f var/tmp/usr1/pools > /dev/null
        then
                # Print the filename if a match is found.  All goes into /tmp/$$
                echo "$FILE"
        fi
done > /tmp/$$


# "echo a b c | xargs ls" is equivalent to "ls a b c".
#
# List all the files in /tmp/$$, sort by time (-t), take only the most
# recent( | head -n 1)
xargs ls -t < /tmp/$$ | head -n 1
# delete the temp file
rm -f /tmp/$$

---------- Post updated at 09:56 AM ---------- Previous update was at 09:52 AM ----------

You could dispense with the temp file completely actually.

for FILE in /usr01/archive/PStat*.Z
do
        # Pipe the decompressed file into grep.  Use a pattern file (-f var/tmp/usr1/pools),
        # match fixed strings instead of regexes (-F), and stop on the
        # first match(-m 1) so scanning the entire file isn't always necessary.
        if zcat "$FILE" | grep -m 1 -F -f var/tmp/usr1/pools > /dev/null
        then
                # Print the filename if a match is found.  All goes into xargs
                echo "$FILE"
        fi
done | xargs ls -t | head -n 1

i've tried the snippet above, but no luck, getting below error

grep: illegal option -- m
Usage: grep -hblcnsviw pattern file . . .

This is why you should always say what your system is in your opening post... Your system doesn't have some options I was using.

What kind of rotten grep doesn't even have -f, though! That's going to make the code many, many times slower...

If your system has egrep, you could try using that in the same code. Otherwise:

for FILE in /usr01/archive/PStat*.Z
do
        zcat "$FILE" > /tmp/$$

        while read STR
        do
                if grep -l "$STR" "/tmp/$$" > /dev/null
                then
                        echo "$FILE"
                        break
                fi
        done < var/tmp/usr1/pools
done | xargs ls -t | head -n 1

rm -f /tmp/$$

you can ignore "-m 1" from the grep command and still use the code.

--ahamed

Worth a shot I guess, but it doesn't say it has -f or -F...

no luck, even i took off "-m 1".
i'am using solaris 8. so, any thing compatable help be greatly appreciated

In what way did it not work when you took off -m 1? Always be descriptive!

Did you try egrep like I suggested?

Have you tried my alternative code?

hi,
i tried the code, it ended up finding the latest file for the last string "ST1604" of /var/tmp/usr1/pools, what about the other strings in the list /var/tmp/usr1/pools

Sorry, I wasn't clear on that -- it only found the latest file with any of those strings in it.

How about this:

for FILE in /usr01/archive/PStat*.Z
do
        zcat "$FILE" > /tmp/$$

        N=0
        while read STR
        do
                grep -l "$STR" "/tmp/$$" > /dev/null &&
                        echo "$FILE" >> /tmp/${N}.$$

                N=`expr $N + 1`
        done < var/tmp/usr1/pools
done

N=0
while read STR
do
        printf "%s:\t" $STR

        xargs ls -t < /tmp/${N}.$$ | head -n 1
        rm -f /tmp/${N}.$$
        N=`expr $N + 1`
done < var/tmp/usr1/pools

rm -f /tmp/$$

/usr/xpg4/bin/grep ...

It still doesn't have the -m options but it has the -f and -F ones.

Andrew

Even working properly, my original script doesn't do what he intended, meaning, the improvements possible with -f and -F aren't relevant anymore :o

Thanks Very Much, it worked this time.

---------- Post updated 09-15-11 at 01:10 PM ---------- Previous update was 09-14-11 at 05:04 PM ----------

Hi Corona688,

when i tried the script with fresh set of strings in "var/tmp/usr1/pools", i'am getting the below error

grep: RE error 41: No remembered search string.

could you please advise as to why the script is not working for other strings?

Without seeing the strings I'm only guessing, but the strings might contain + ? * . [ ] { } ( ) \ characters that grep would consider an expression, not literal characters. So you turn out to need -F after all (but not -f).

for FILE in /usr01/archive/PStat*.Z
do
        zcat "$FILE" > /tmp/$$

        N=0
        while read STR
        do
                /usr/xpg4/bin/grep -F -l "$STR" "/tmp/$$" > /dev/null &&
                        echo "$FILE" >> /tmp/${N}.$$

                N=`expr $N + 1`
        done < var/tmp/usr1/pools
done

N=0
while read STR
do
        printf "%s:\t" $STR

        xargs ls -t < /tmp/${N}.$$ | head -n 1
        rm -f /tmp/${N}.$$
        N=`expr $N + 1`
done < var/tmp/usr1/pools

rm -f /tmp/$$

Hi corona688,

i'am using the above code exactly the same for zipped files in different location (example: "for FILE in/usr02/archive/mf_pstat*.Z") of the filesystem using data in "/var/tmp/usr1/pools" having data like below

461719
461723
461777
461778
461779
461780
461781

when i run the script using above i'am getting the below response from the script.

461719: ./mfpoolex.sh: /tmp/0.18553: cannot open
461723: ./mfpoolex.sh: /tmp/1.18553: cannot open
461777: ./mfpoolex.sh: /tmp/2.18553: cannot open
461778: ./mfpoolex.sh: /tmp/3.18553: cannot open
461779: ./mfpoolex.sh: /tmp/4.18553: cannot open
461780: ./mfpoolex.sh: /tmp/5.18553: cannot open
461781: ./mfpoolex.sh: /tmp/6.18553: cannot open

not sure whether the files in /tmp are even getting created. please help.

thanks