moving large number of files

Shell Programming and Scripting
1

I have a task to move more than 35000 files every two hours, from the same directory to another directory based on a file that has the list of filenames

I tried the following logics

(1)
find . -name \*.dat > list

for i in `cat list` do mv $i test/ done

(2)
cat list|xargs -i mv "{}" test/

logic 1 takes 40mins and logic 2 in 25 mins

Is there any other faster way I can move files based on a file that has the list of filenames.

2

Are you sure you need the list?

mv *.dat test

P.S. May be tar/cpio are better for large number of files ...

3

I don't know that this would be faster, but it should execute "mv" fewer times
(You'd have to figure out an appropriate number for -s)

xargs -s 1000 echo  < list | sed 's+^+mv +;s+$+ test/+' |sh -

If you have perl and the test directory is in the same partition, you might try

perl -nle 'rename "$_", "test/$_";' list
4

Just a suggestion..... Not sure if this is even an option for your environment.

Don't move the files first. Just create links from the old dir to the new dir. run the link command first. then come back and move the files to over write the link. this way the files are available very quickly and your move cmd can take 40 minutes.

Just thinking outside the box.

5

I doubt this command, because OP had said 35000 files and more.

What if the number of command line arguments to ' mv ' exceeds the permissible limit ?

Its quite possible ! :slight_smile:

6

Here is another way,

with "list" as the file with list of filenames

split the list file into n chunks .. list_1, list_2 ... list_n

you should arrive at a bench mark say 10 to 50 or something like that

now operate any of the logic on the n list_* files simulatenously

and this should definitely be faster as mv is executed in multiples. :slight_smile:

7

Didn't think about that :slight_smile:

So, with recent zsh:

setopt nomatch
while set -- *.dat([1]);do
mv -- *.dat([1,1000]) test
done

Adjust 1000 for your limit.

8

Bryan,
Whenever an issue becomes difficult to resolve, it is best to analyze
the entire context of the problem.

Could you please tell us all relevant information regarding your problem:

1) What is the frequency files are added to the directory?
Example: Constantly, every minute, every hour, etc.

2) Is there a rule in the formation of the file name?
Example: All files start with "ABC", then a sequential number, etc.

3) In the average, how big is each file?

4) Is there any other files in this directory besides the files to be moved?

After you answer these questions, we will have a more definitive solution
for your issue.