Copying files

Shell Programming and Scripting
1

All,

I need to grab and rename common files from several unique directory structures. For example, the directory structures looks like:

/unique_dir/common/common/common/person_name_dir/common_file.txt

There are over 90,000 of these text files that I'd like to put in a single directory as person_name_common_file.txt. I tried to write a simple perl script but I'm a bit over my head:

#!/usr/bin/perl

use File::Copy;

my $dir;
my @dir = `find . -type d -mindepth 4`;

chdir "/home/admin/pack";

foreach $dir (@dir) {

        chomp $dir;
        move ($dir,"/home/admin/out/");

}

exit 0;

This partially works for the first step of putting all the person_name_dir into a new folder but I need to consider duplicate names (which name1, name2, name3 would be fine) and doing the pre-pending of the common text file with the person's name.

Hopefully I haven't confused my intentions too much. If this is easier done in a shell script or other please let me know. Any guidance is greatly appreciated.

Thanks in advance,

Herb

2

It's a very bad idea to put 90,000 files in one directory. Creating or deleting files inside that folder will become very slow.

What is your actual goal here? I don't mean "moving the files". Why are you moving the files? If we can find a way to accomplish your ask without dumping 90,000 files in one directory you'll be much better off.

1 Like
3

The files will be used as a one time read of the txt files. There will not be ongoing reads/writes.

4

Can't you do this

 mkdir ("/home/admin/out/person_name_dir");
 move ($dir,"/home/admin/
out/person_name_dir");

Wouldn't it solve the performance problem and the name conflicts?

5

That is not the problem, though:

This could happen during the move, because moving a file counts. Anything which alters the output of 'ls' is altering the directory entry, which is what slows down. Moving the first few thousand files will happen lickety-split, but it will quickly bog down. It could take hours or days to move them all. That's why those files have been split into a tree of folders, to avoid this. All for an operation you say you're going to be doing what -- once? What will you do with them after -- leave them sitting forever, or delete them? Deleting that massive pile of files will take a similarly awful amount of time.

Strangely, "ongoing reads/writes" usually aren't a problem. Creating one file isn't so bad... Opening one file isn't so bad... Which is how people end up with folders full of 90,000 files -- something logging or recording for years and years without cease in the same place. Someday they need to delete them all, and find themselves in big trouble. We regularly get threads like this.

So, let me repeat my question:

Please tell us why you think you need 90,000 files in one place. What could you do with them all in one place, that you can't do now?

6

@Corona688, thank you for your replies. I've gone back to the team that asked for the initial copy and asked them why they want to do what they're asking.

They are re-thinking what they need and I'll update if necessary.

Thanks again,

Herb