Copy Directory Tree from one Server to Another

Hi,

I have a serverA with Directory "/opt/app/myfiles " which is 8.5GB with more than 40000 files and 45 folders in it.

I wish to transfer the folder " /opt/app/myfiles " tree structure with all the sub-directories and skip all files to another serverB.

I would appreciate a command that is Quick and I would not mind if the directory structure is copied locally so I manually transfer and untar it at the destination.

Note: In any case my source serverA Directory " /opt/app/myfiles " should not be harmed in any way.

You want empty directories on serverB.

echo 'mkdir /opt/app/myfiles' > mkdirs.shl
find  /opt/app/myfiles -type d |
while read rec
do
  echo "mkdir $rec"
done >> mkdirs.shl
chmod +mkdirs.shl
scp mkdirs.shl serverB:
ssh serverB './mkdirs.shl'

You see this works but it takes more than an hour for the find command as it is affected by the thousands of files which are present in the folder. Any better fix ?

I might suggest a sort after the find then pipe to xargs -n1 mkdir to ensure that directories are created in ascending order so that there is no risk of trying to create /a/b/c/d/e before you create /a/b/c I admit it is unlikely, but possible.

if you are using servers that use backup & restore, fsdump & fsrestore or ufsdump and ufsrestore then you can take a dump and I think there is a way using the interactive restore to generate all the directories ready for data restore, at which point you just cancel. It's been a long time since I used them though. They are on AIX, Solaris etc. I'm not sure if there is a Linux variety.

Robin

Hi.

I may be missing some important point, but I see:

1)

I tried the find in the following situations:

Starting directory "/usr"
 Found directories: 9536
 Found files      : 125144
 Mean files/dir   : 13.12
 Size (bytes)     : 3.0G

real	0m2.607s
user	0m0.472s
sys	0m2.064s

and

Starting directory "VirtualBox VMs/" (different computer)
 Found directories: 56
 Found files      : 163
 Mean files/dir   : 2.91
 Size (bytes)     : 277G

real	0m0.062s
user	0m0.008s
sys	0m0.016s

and on the very slow ZFS fielsystem (Linux SuSE):

Starting directory "/usr" (in a VM)
 Found directories: 8987
 Found files      : 119022
 Mean files/dir   : 13.24
 Size (bytes)     : 3.8G

real	1m19.483s
user	0m1.855s
sys	0m25.096s

Still, the file and directory names (not size) were listed for ZFS in:

real	0m7.468s
user	0m1.000s
sys	0m4.339s

and to write the directory names to a file 3.143s

While I have not created 40K files in the 277G directory, it still seems odd that the operation would an take an hour.

2) Although individual directories can be made with mkdir, the command also allows multiple parameters, so

mkdir path1 path2 path3 ...

could be used.

3) If the parents options is used, then the creation order is irrelevant (as far as I can tell):

#!/usr/bin/env bash

# @(#) s1	Demonstrate mkdir "parents" option.

pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C mkdir tree

# Remove debris from previous runs.
rm -rf a

# Create a tree.
pl " Create a tree with "parents" mode:"
mkdir -p a/b/c/d/e
tree -n -d a

# Create a sub-tree.
pl " Create a sub-tree with "parents" mode -- expecting no change:"
mkdir -p a/b/c
tree -n -d a

exit

producing:

$ ./s1

Environment: LC_ALL = , LANG = en_US.UTF-8
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian 5.0.8 (lenny, workstation) 
bash GNU bash 3.2.39
mkdir (GNU coreutils) 6.10
tree v1.5.2 (c) 1996 - 2008 by Steve Baker, Thomas Moore, Francesc Rocher, Kyosuke Tokoro 

-----
 Create a tree with parents mode:
a
`-- b
    `-- c
        `-- d
            `-- e

4 directories

-----
 Create a sub-tree with parents mode -- expecting no change:
a
`-- b
    `-- c
        `-- d
            `-- e

4 directories

Best wishes ... cheers, drl