Recursive copy of Folders with files

Dear All,

I will appreciate any help received. Our system is running on hpux v1

My problem is as follows:

We have many customer folders with name fd000100 , fd000101 and so on
e.g.

(Testrun)(testsqa):/>ll /TESTrun/fd000100
total 48
drwxrwx---   2 fq000100   test         96 Jun 27  2004 ca
drwxrwx---   2 fq000100   test         96 Jun  1  2003 ipo
drwxrwx---  761 fq000100   test      20480 Aug 27 14:36 reports
drwxrwx---   2 fq000100   test         96 Jun  1  2003 settle
drwxrwx---   2 fq000100   test       4096 Feb 17  2013 sign
(Testrun)(testsqa):/>

Each customer folder has 5 sub-folders. Our target folder is reports folder.
This reports folder has many sub-folders and many files in each sub-folders.

We need to copy these reports folders with its parent folder e.g. fd000100 along with all sub-folders in it.

Say our destination path is /Testrun/new-dest

So, after copy the listing should be as follows:

(Testrun)(testsqa):/>ll /TESTrun/new-dest/fd000100
total 48
drwxrwx---  761 fq000100   test      20480 Aug 27 14:36 reports
(Testrun)(testsqa):/>

(Testrun)(testsqa):/>ll /TESTrun/new-dest/fd000101
total 48
drwxrwx---  761 fq000100   test      20480 Aug 27 14:36 reports
(Testrun)(testsqa):/>

and so on...

Any help or hints on this recursive copy command will be highly appreciated.

Thanks you in advance.

Hi,

Not entirely sure what you want to do here, am I to understand that you want to copy everything to a new destination - and place the other four directories inside the reports directory.

Or am I missng something here.

Regards

dave

you may use cp command with -rP optins (r for recursive; P for directory)

cp -rP fd000100 /TESTrun/new-dest/ 

---------- Post updated at 06:20 AM ---------- Previous update was at 06:18 AM ----------

if you already have the directory in destination folder, you may use -R option to merge the directories

In the TESTrun directory, try this manually to see if it works in general as expected:

tar cvf - fd000100/reports | (cd new-dest && tar xvf -)

If the result is OK, and assuming all directories starting with fd do not contain any whitespaces, you can do the rest via for loop:

for i in fd*; do tar cvf - $i/reports | (cd new-dest && tar xvf -); done

Why not

tar cvf - fd*/reports | { cd /TESTrun/ && tar xvf - ; }

(if it fits onto the command line)?

We have around 700 such fd000100 folders e.g. fd000101, fd000102, fd000103 and so on. Hence manual copy is not an option. I tried

cp -rp fd*/reports /Testrun

But it does not serve my purposes. In the destination path the fd000100 folders are not present. Is it possible to create these folders while copying files?

The source directory structure is:

RUNenv/fd000100/reports/01-JAN-2014/abc.txt
RUNenv/fd000100/reports/01-JAN-2014/def.txt
.
.
.
RUNenv/fd000100/reports/02-JAN-2014/xyz.txt
.
.

There are few millions of such text files.

I am searching for a solution to copy these directory recursively with sub-folders and files to a new destination such as

/TESTrun

So, after copy the new directory should be as follows

TESTrun/fd000100/reports/01-JAN-2014/abc.txt
TESTrun/fd000100/reports/01-JAN-2014/def.txt
.
.
TESTrun/fd000100/reports/02-JAN-2014/xyz.txt

Well its possible to create these folder before copying files into... It means using a loop...
e.g. Using Junior-helper's code:

newdest=<where to copy>
for i in fd* 
do 
   mkdir $newdest/$i
   tar cvf - $i/reports | (cd $newdest && tar xvf -)
done

Dont ask if the code works, because I tested... But if you have issues its because you did not say all e.g. who are the owners of the dirs and files? tar will preserve them but what about $newdest/$i ?
We cant guess all.. it looks like those directories have the same name as their owner in which case you will have to do and chown and even perhaps a chmod because I cant guess what umask you have, if any...

Thanks a lot guys

tar cvf - fd*/reports | { cd /TESTrun/ && tar xvf - ; }

This command worked fine.

But, the ownership and permission changed after copy. How to keep it same as source files?

If you ran that as a normal user:
man tar :

And you have been told so in post#7...

cp -rp creates directories and subdirectories

That's true, in general, but it's not applicable here. I think you got a wrong idea of mhbd's plan.

If we concentrate on one customer only, then it's not simply

cp -rp fd000100 /TESTrun/new-dest/

it's more like

cp -rp fd000100/reports /TESTrun/new-dest/

If you run the latter, the folder reports is copied to new-dest , but then you don't know which customer do those reports belong to.
The result should be new-dest/fd000100/reports

A work-around could be to manually create the new-dest/fd000100 folder before copying like so

cp -rp fd000100/reports /TESTrun/new-dest/fd000100

but manually created (or restored with non-root?) folder raises another issues, like vbe has forseen correctly.

sorry, I figured out now.

finally

Hello Guys, thanks for your comments and suggestions.

Here is the current situation. I tried the command mentioned below in our test server which has same

folders structure as LIVE:

#tar cvpf - fd*/reports | { cd test-copy/ && tar xvpf - ; }

It worked fine. It copied all folders and sub-folders with files with original owner & permission at

once in a signle command. But we don't want to run this command in our LIVE server as the folders & files size is 1200GB involving millions of files and it may fail.

So, I tried to copy folders & files year-wise with following command:

#tar cvpf - fd*/reports/*2003* | { cd test-copy/ && tar xvpf - ; }

It copied all folders and sub-folders with files of year 2003, size is matching with source but owner &

permission changed i.e. not same as source. It seems to me that the tar command could not keep owner &

permission same as original when I put 2 arguments in the command one is fd* and another is /*2003*

I searched man tar but no clue found.

Note: each parent folder's owner is different. I am running the tar command from root user.

Our source directories are as follows:

/SQArun>ls -ltr fd000001 fd000010 fd000011

fd000010:
total 88
drwxrwx---   2 fq000010   dsqarun      24576 Jul 20 14:24 bills
drwxrwx---  759 fq000010   dsqarun      20480 Aug 27 13:01 reports

fd000011:
total 72
drwxrwx---   2 fq000011   dsqarun      16384 Sep 10  2013 bills
drwxrwx---  760 fq000011   dsqarun      20480 Aug 27 14:36 reports

fd000001:
total 104
drwxrwx---   2 fq000001   dsqarun      32768 Jul 20 14:24 bills
drwxrwx---  760 fq000001   dsqarun      20480 Aug 27 13:01 reports
/SQArun>

There are many fd* folders which generated sequentially and total 800+ folders. 

/SQArun/fd000001/reports>ls -ltr -d *2003*
drwxrwx---   2 fq000001   dsqarun         96 May 31  2003 01-31-MAY-2003
drwxrwx---   2 fq000001   dsqarun         96 Jun  2  2003 01-01-JUN-2003
drwxrwx---   2 fq000001   dsqarun         96 Jun  2  2003 01-02-JUN-2003
drwxrwx---   2 fq000001   dsqarun         96 Jun  4  2003 01-03-JUN-2003
drwxrwx---   2 fq000001   dsqarun         96 Jun  4  2003 01-04-JUN-2003
drwxrwx---   2 fq000001   dsqarun         96 Jun  9  2003 01-05-JUN-2003
drwxrwx---   2 fq000001   dsqarun         96 Jun  9  2003 01-07-JUN-2003
drwxrwx---   2 fq000001   dsqarun         96 Jun  9  2003 01-08-JUN-2003
drwxrwx---   2 fq000001   dsqarun         96 Jun  9  2003 01-09-JUN-2003
drwxrwx---   2 fq000001   dsqarun         96 Jun 10  2003 01-10-JUN-2003
drwxrwx---   2 fq000001   dsqarun         96 Jun 11  2003 01-11-JUN-2003
drwxrwx---   2 fq000001   dsqarun         96 Jun 12  2003 01-12-JUN-2003
drwxrwx---   2 fq000001   dsqarun         96 Jun 14  2003 01-14-JUN-2003
drwxrwx---   2 fq000001   dsqarun         96 Jun 15  2003 01-15-JUN-2003
.
.
.
/SQArun/fd000001/reports>

In this fd000001 folder there are many such date folders where last 4 digits is year and ranging from year 2003 to 2014

Any further suggestions will be appreciated.

Thanks

Why don't you run it in chunks as tar cvpf - fd000{001..050}/reports

Hi RudiC, thanks for the tips.

I tried this to try with 2 folders (fd000010 & fd000011)

#tar cvpf - fd000{010..011}/reports | { cd test-copy/ && tar xvpf - ; }

But it is giving error

/SQArun>tar cvpf - fd000{010..011}/reports | { cd test-copy/ && tar xvpf - ; }
tar: cannot open fd000{010..011}/reports
/SQArun/test-copy>cd

Too bad - that's a brace pattern available in recent shells. Try setting the braceexpand (-B) option (ksh).

You could try it with filename expansion this way:

#following should match directories fd000100 to fd000109
tar cvpf - fd00010?/reports | { cd test-copy/ && tar xvpf - ;}

#following should match directories fd000100 to fd000199
tar cvpf - fd0001??/reports | { cd test-copy/ && tar xvpf - ; }

If the above fails, you could try it with this approach for running it in chunks:

#first, create a temporary list with all relevant directories:
$ ls -d fd* > dirlist

#get first two entries (fd000100 & fd000101)
$ awk 'NR==1,NR==2' dirlist
fd000100
fd000101
$

#combine it with tar command 
$ awk 'NR==1,NR==2' dirlist | while read dir
 do
 tar cvpf - $dir/reports | { cd test-copy/ && tar xvpf - ; }
 done

If it works, then you take bigger chunks, eg. 'NR==1,NR==100', then 'NR==101,NR==200' and so on.