Help with zipping files

Shell Programming and Scripting
1

Hi,

I have come across a requirement in which I need to zip files. This is fine but the requriement has one conditions like below:

One .z file can not have more than 10,000 files

Now in the directory I have several files liek below:

aaa_file_10_00001.txt
aaa_file_10_00002.txt
aaa_file_20_00001.txt
aaa_file_30_00001.txt
aaa_file_10_00003.txt
...........................
...........................
...........................
aaa_file_30_00750.txt
aaa_file_30_10005.txt

If i have to zip the above files, I will have 4 .z files. One for files like aaa_file_20_.txt, one for files like aaa_file_30_.txt and two for files like aaa_file_10_.txt I am planning to move files of different type (diferentiated by 3rd field in the file name) to a temporary directory and zip them. But the problem is I can not zip more than 10,000 files together. How do I move files of type aaa_file_10_.txt from 1-10000 (this is the value in 4th field of the file name )first, zip them and then move the reamining 10001-10005.

Using a loop is not a good option as per my understanding. Is there any other way of doing it which will not have impact on performance ?

Thanks
Angshuman

2

In bash,

for i in {00000..10000}
do
 echo aaa_file_10_${i}.txt ## take whatever action you want to perform (eg moving)
done

you can also use 'seq' command or even just normal incremental count.

3

Hi Anchal,

Thank you for your reply. I understand that we can achieve this using loop. But my concern is performance. If I use a loop when there is actually more than 10,000 files of same type, it will take lot of time. I was thinking if there is any one liner using awk or any other way if I can do that.

Thanks
Angsuman

---------- Post updated at 08:33 PM ---------- Previous update was at 03:51 PM ----------

Hi All,

Any one can help on this?

Thanks
Angsuman

4

Any one can help me on this?

5

You agreed not to bump posts when you registered; nobody here is "on call". If you really, really need help RIGHT NOW there's even an "emergency support" forum you didn't use too.

It would help a lot to know what your system and shell is.

---------- Post updated at 12:13 PM ---------- Previous update was at 11:47 AM ----------

Run this in its own empty directory to see what it does. It should work in most shells. You can probably adapt it to do what you want.

#!/bin/sh

# Create some example files to work on.
# it will create directories 00 through 09.
# 00/ will contain files 00-00 through 00-99,
# 01/ will contain files 01-00 through 01-99, etc.
for DIR in 00 01 02 03 04 05 06 07 08 09
do
        mkdir -p $DIR
        for A in 0 1 2 3 4 5 6 7 8 9
        do
                for B in 0 1 2 3 4 5 6 7 8 9
                do
                        touch "$DIR/$DIR-$A$B"
                done
        done
done

# Let's list all the 00-* files, and split the list apart at 20 lines each.
# The 'split' command will create files "list-aa", "list-ab", and so forth.
# the 'sort' makes the filenames come out in order from 00 to 99, if
# you don't care about that you can leave it out.
find ./ -type f -name "00-*" | sort | split -l 20 - list-
# Now we use each list to create its own zip file.
for LIST in list-*
do
        ZIPFILE="${LIST}.zip"
        zip -r "${ZIPFILE}" -@ < "${LIST}" && rm "${LIST}"
done

# show our zipfiles
ls *.zip
# how many files in the first zip?
unzip -l list-aa.zip
6

performance issues? zipping 10,000 files will never be quick.
maybe in 2015 it will.

since when is a loop slow? it's all the zipping which will
be the bottleneck. not any loop.

you can use shell expansion if you know the file names, and they adhere
to a numbering sequence

mv blah_[0][0-9][0-9][0-9][[0-9].txt

no loop,
hmm, but I wonder if internally bash would express the above as a loop?

or, you could list the names to a file, and use split(1) to get separate lists of names.

7

Most sane shells would probably express it as a "too many arguments" error...