Looping inside directories based on a file which contains file directory list

Shell Programming and Scripting
1

Hi All,

Please help.
I have got a file which contains a listing of a file and some directories after it, one by one. I am calling this file xyz.txt here

file1
dir1
dir2
dir3
dir4
file2
dir5
dir6
dir7
dir8
file3
dir9
dir10
dir11
dir12

Requirement is that I have to pick the files one by one from xyz.txt file i.e. file1 then file2 then file3 ... for file1 I have to go inside directories listed below file1 i.e. dir1, dir2,dir3, dir4. for file2 I have to go inside directories listed below file2 i.e. dir5,dir6, dir7, dir8. means like a loop. Actually, inside these directories there are some log files. i have to search for a pattern (some records)from these log files and grep these records from the files mentioned in the file xyz.txt ( this part looks easy and i can try for it).
Only thing is I am not able to loop through file xyz.txt

thanks.

2

Not clear. How do you tell dirs from files? Always four dirs? Where do you get the patterns to grep for? Be more specific. What have you tried so far?

3

Thanks for the reply. Sorry if i am not clear here.
suppose file1 is as follow ( and other files file2, file3 etc also similar to this, only data is different)

00020856820000      00544     00000000000001028
00068826707655      00155     00000000000006013
00003150631309      00166     00000000000002027
00004800121348      00088     00000000000004042
00068826704470      00645     00000000000001101
00001380055665      06005     00000000000001045
00078616200151      00603     00000000000004014
00002880015010      06111     00000000000001099
00004400002795      00603     00000000000006142
00002073511018      06078     00000000000004044
00002073509273      06078     00000000000001024
00000000004329      00144     00000000000002133
00004152087031      00666     00000000000001080
00004470000927      00777     00000000000001106
00002310010266      00066     00000000000002129
00001251140600      00650     00000000000001116
00004180050130      00666     00000000000002036
00007218056750      00444     00000000000002049
00068826712549      00650     00000000000001027
00004140900410      00767     00000000000002134

dir1, dir2, dir3,dir4 contain the logs, log1.txt,log2.txt,log3.txt
log1.txt is inside dir1
log2.txt is inside dir2
log3.txt is inside dir3
log4.txt is inside dir4

dir1,dir2,dir3 etc names are always like this, i.e always word "dir" comes at the starting in directory name.

suppose log1.txt is

00068826707655 sfdfsdfsf
00003150631309 gdgdgdgdg
00004800121348 sgsgsgsgsg

log2.txt is

00004400002795 gsgsgsgsgsg
00002073511018 gsgsggsgsgg

log3.txt is

00007218056750 gsgsgsgsg
00068826712549 gsgsgsgsg
00004140900410 gsgggsgsg

log4.txt is

00001251140600 fasdfasfs

so i am thinking like:

1) Traverse file xyz.txt
2) First, read file1, store the filename(file1) in some variable
3) Traverse through dir1,dir2,dir3,dir4 and cut the first field from 1-14 ( only the numbers) from the logs log1.txt, log2.txt, log3.txt, log4.txt i.e.

00068826707655
00003150631309
00004800121348
00004400002795
00002073511018
00007218056750
00068826712549
00004140900410
00001251140600

4) grep these numbers from the file1 and make a new file.

00068826707655      00155     00000000000006013
00003150631309      00166     00000000000002027
00004800121348      00088     00000000000004042
00004400002795      00603     00000000000006142
00002073511018      06078     00000000000004044
00007218056750      00444     00000000000002049
00068826712549      00650     00000000000001027
00004140900410      00767     00000000000002134

5) Then, repeat the same steps from 2) to 4) for file2

6) Then, repeat the same steps from 2) to 4) for file3

4

Well, not knowing your system nor shell (which you did not mention), based on your examples and requirements, and hoping I understood and interpreted everything correctly, I've come up with this, which works on linux and bash:

#!/bin/bash
# set -vx
function grepit () {
                  eval $DIRS" | cut -d\" \" -f1 > pattfile"
                  grep -f pattfile $fn >$fn.grep
                 }

FIRST=1
DIRS="cat "
while read line
     do if [ "${line:0:3}" != "dir" ]
            then    [ "$FIRST" -eq "0" ] && grepit
                    fn=$line
                    FIRST=0
                    DIRS="cat "
            else    DIRS=$DIRS" "$line"/log*.txt"
       fi
     done <xyz.txt
grepit
rm pattfile

Results will finish up in respective "filename.grep". Pls. test and come back with results.

5

Thank You RudiC for all your efforts.
Sorry I forgot to mention system and shell. :frowning:
system is AIX 6.1 and shell is ksh
I will try your code.

in your code

 do if [ "${line:0:3}" != "dir" ]

i am not sure if this line of code will work in ksh.
i guess you are looking for 4 directories always. In my case it could be any number of directories and not 4 always ... You mentioned already " Always four dirs?" i missed it. very sorry for that :frowning:

---------- Post updated 10-17-12 at 01:35 PM ---------- Previous update was 10-16-12 at 09:26 PM ----------

I tried to run your script by changing the code from

#!/bin/bash

to

#!/bin/ksh

and removing that "function" keyword, but it failed in this line

"${line:0:3}": bad substitution

I guess this doesn't work in ksh.

6

My little meek script will work for more than four directories. In that if clause you'll need to tell files from directories, that's all. Sorry I don't have access to a ksh shell. In bash you could do a check for directory like [ -d "$line" ] or file like [ -f "$line" ] ; according to the man page, this should exist in ksh, or, try to find a ksh counterpart.

7

thank you very much Rudic. I will check.