Unzip in bash script

My script works like this.

1. First for loop - checks if it can access zip parameters in a directory
2. If unzip fails for any of the file in that directory, then it goes inside the second for loop - to list which file is that
3. My doubt is..Is there any way I can tell status for every 5000 files in the first for loop. like processed 5000 files...10000 files etc...for monitoring
   4.I can very well keep a counter and do this logic in second for loop, since it loops every file.but this will execute only when there are corrupted files

for j in `find $i -name "abc*" -type d`; do
              unzip -qT $j/\* 2>>/dev/null;
             if [ $? -ne 0 ]; then
                for k in `find $j -type f`; do
                   unzip -qT $k 2>>/dev/null;
                done
             fi
done

Any inputs pls...

Thanks

At the first line of your loop :

((i++)); (($i%5000)) || echo "$i files processed"

Frans,

for j in `find $i -name "abc*" -type d`; do 
// Here $i is directory.
((i++)); (($i%5000)) || echo "$i files processed"
// Here it will display the count of sub-directories. I need to display list of
files in sub-directories.

Pls help.

Thanks

Apologize : Replace i with another unused letter !
I think you could try to avoid the first check (unzip -qT $j/\*), believe that it should'nt slow-down the script to much and you don't need to look twice if there is an error with unzip.
I suggest (some modif. to make it more readable)

for FILE in $(ls abc*/*) # This should work
do
    ((n++)); (($n%5000)) || echo "$n files processed"
    unzip -qT $FILE 2>>/dev/null || echo "Error in file $FILE"
done

I get error saying all the files are corrupted.

 for FILE in $(ls $j) # This should work
// $j - last level subdirectory which has list of zipfiles without .zip extension
do
   ((n++)); (($n%5000)) || echo "$n files processed" 
    
    unzip -qT $FILE 2>>/dev/null 
    if [ $? -ne 0 ]; then
           echo $FILE is BAD 
    fi
   
done

Instead if I use as I mentioned in the below code it gives the correct output.

 for k in `find $j -type f`; do

performance :

if I run the code with the 2 loops - 1. scan all the files in the directory with one unzip system call
2. If any of the dir has corrupted files, loop the second directory and list the files
It takes 0m5.186s this time to execute 1.3 lakhs files

If i remove the outer loop, it takes 1m46.280s to execute 1.3 lakh files

Our server will hold data close to 1.4 TB. There is a performance hit in these 2 ways of implementation. Thats the reason, I wanted to count the files in the outer for loop (while testing the unzip for all the files in one sub-directory)

Frans, your help pls.

Thanks,
Vidhya.

Can you tell me how many directories and how many files you have in each one.
a command like

(( n+=$(find $j -type f | wc -l) ))

would increment n by the number of files in the dir.
Please put code tags around code

Not only is ls unnecessary, but it will break your script if any filenames contain whitespace or other pathological characters.

Use wildcard expansion directly:

for FILE in abc*/*

I ll have minimum 15 main directories under each there are 10 -15 sub-directories

Each subdirectory will hold minimum 10000 files.

Thanks,
Vidhya

You can only count yhe number of files processed in each subdir.

for j in $(find $i -name "abc*" -type d)
do
    (( n+=$(find $j -type f | wc -l) ))
    if ! unzip -qT $j/\* 2>>/dev/null
    then
        for k in $(find $j -type f)
        do
            unzip -qT $k 2>>/dev/null;
        done
    fi
    echo "$n files processed"
done

Fran,

Yeah ...only a miminal information - total files in each sub-directory can be given.

Thanks for your inputs the above script solves my problem.

Not only is find unnecessary, but it will break your script if any directory names contain whitespace or other pathological characters.

Use wildcard expansion directly:

for j in abc/*

That syntax is not portable; use the POSIX syntax:

n=$(( n + $(find $j -type f | wc -l) ))

However, you don't need find or wc; it can be done in the shell itself:

set -- *
n=$(( $n + $# ))

Not only is find unnecessary, but it will break your script if any filenames contain whitespace or other pathological characters.

Johnson,

Thanks for your corrections. Im a beginner and your inputs helped me to learn more stuff

I tried checking in few tutorials, but i didnt understand.Can you please clarify me on the behaviour of two lines of code

  set -- *
n=$(( $n + $# ))
 

Thanks,
Vidhya

set -- *

sets the filelist (with wildcard * but could be i.e. *.txt) as the parameters $1, $2 ...

n=$(( $n + $# ))

$# returns the number of parameters --> the number of files in this case, and increment n by that number

Thanks frans
Below is my final code draft

 
#!/bin/bash
bfile=/tmp/BAD_BITFILES_`hostname`;
for i in /store/archive/TP_* ; do
   echo "Processing main directory : $i"
      for j in $i/L*/L*/L2*; do
         echo $j
         # n=$(( n + $(find $j -type f | wc -l) )) - statement 1
         set -- *
        n=$(( $n + $# ))   - statement 2
         unzip -qT $j/\* 2>>/dev/null;
         if [ $? -ne 0 ]; then
             for k in $j; do
                  unzip -qT $k 2>>/dev/null;
                  if [ $? -ne 0 ]; then
                      echo $k is BAD;
                      echo $k >> $bfile;
                  fi
              done
         fi
    done
echo $n
done
 

If i replace stmnt 1 with stmnt 2 (highlighted in brown color in the above code)as suggested by johnson, the output varies

If I use stmnt 1 the output is and this is the right count of files

If I use stmnt 2the output is wrong

Any help pls..

Thanks
Vidhya.

Of course, your file selection is $i/L*/L*/L2* so you should : (modifications in dark red). I use the variable 'files' because it is used 2 times.

files=$i/L*/L*/L2*
for j in $files; do
         echo $j
         # n=$(( n + $(find $j -type f | wc -l) )) - statement 1
         set -- $files
        n=$(( $n + $# ))   - statement 2
...

after modification too its giving wrong result.

8222 is the expected result.But it outputs the value as 5690

I believe it's then better to use the first statement (with find and wc).

Yes.. Thank you.