Moving 100K file to another folder using 1 command

Shell Programming and Scripting
1

Hi,

I need to move 1000s of files from one folder to another.
Actually there are 100K+ files.
Source dir : source1
Target dir : target1

Now if try cp or mv commands I am getting an error message : Argument List too long.

I tried to do it by the time the files are created in the source folder.

-rw-r--r-- 1 prod usr1 104 Dec 3 08:01 200912_011.txt

For example the above file we can see that files can be seggregated based on the timings - 08:01 > 1000 file
08:02 > 1000 file.
I can select the time from the ls -lrt command using awk '{print $8}'.

But not able to move files writing a command line. This is how I tried, any solution is welcome.

Thanks in advance.

2

Use acombination with find like for example

find . -type f -name "*" -exec cp {} /targetdir \;

or

find . | cpio -dumpv /targetdir
3

Hello unx100

Good ol' tar is marvellous to copy files from one directory to another:

cd sourcedir
tar -cf - ./ | ( cd targetdir ; tar -xf - )

When writing backup, tar can send data to stdout when you use a single - (hyphen), instead of setting filename for backup file.

On the other hand, when reading, tar can do the same way.

The parentheses ensure the shell enable to shift working directory before writing to working dir.

Have a good time!

4

Bit heavy handed perhaps but I was at a bit of a loose end...

#!/usr/bin/perl
use POSIX;
use File::Copy;

$srcdir = '/source1';
$dstdir = '/target1';

opendir(DIR, "$srcdir");
@FILES= readdir(DIR);
foreach $file (@FILES) {
        chomp $file;

        print "current file is: **$file** \n";
        next unless -f $file;
        copy($srcdir . "/" . $file, $dstdir . "/" . $file) or die "File cannot be copied.";
        }

Specifically looking only to copy files...

The one command would be ./script.pl :slight_smile:

5

Hi.

Unless the files are being transferred to a different filesystem, I would recommend mv so that you are not touching the data inside the files, but rather only manipulating directory entries, a far less intensive activity. If different filesystems are involved, mv will arrange a copy, if I recall correctly.

The command

xargs - build and execute command lines from standard input

was invented to deal with the issue of long argument lists. See man xargs, search the forums, and experiment to see how it will address your problem.

Best wishes ... cheers, drl

6

You could use rsync which is very versatile and adapted for a huge amount of data.

7

Thank you all for your suggestions.