File copy script

Looking for help on a script that does the following. Reads a text file with ~5000 lines that look something like this:

/folder/path/path2/pathX/file.lua,d/1.2699.gz
/folder/path/path4/pathX/file2.lua,d/1.2699.gz
/folder/path/path2/pathX/data/file3.lua,d/1.2699.gz

And copies the file from the following path (/mnt/cifs appended) to the above path:

/mnt/cifs/folder/path/path2/pathX/file.lua,d/1.2699.gz
/mnt/cifs/folder/path/path4/pathX/file2.lua,d/1.2699.gz
/mnt/cifs/folder/path/path2/pathX/data/file3.lua,d/1.2699.gz

In each of these, the following are 0755 folders that may or may not exist:

/folder/path/path2/pathX/file.lua,d/
/folder/path/path4/pathX/file2.lua,d/
/folder/path/path2/pathX/data/file3.lua,d/

And each copied file should be chown�d user:group.

I have a script like working for a specific file value (e.g. 1.2699.gz across multiple directories), but wanted to get this working across multiple files in multiple directories read from a text file.

Thanks!

xargs -n1 -I% sh -c 'test -f % && mv /mnt/cifs/% %' < infile

--ahamed

So what is your script that is working for a specific file?

What shell are you using?

The script ahamed101 provided may work for you, but it won't create missing target directories and it won't set the owner and group of the destination files correctly if the desired owner and group don't match the effective UID and GID of the process running the script (assuming you change the mv to cp ).

Oops!
Yes, I didn't notice the requirement of creating the missing directories.

--ahamed

I take the file missing error log, parse it down to a simple list of files called cleanerror.txt. And run a script that looks like this to restore the files:

#!/bin/bash

file='1.2699.gz'
files=`cat cleanerror.txt`
filesdir=`sed -e 's/$file//g' cleanerror.txt`

for i in $filesdir; do
mkdir -p $i
done

for i in $files; do
cp "/mnt/cifs/$i" $i
chown user:group $i
done

Using bash (running CentOS6) but can use Perl. Wrote this just now, it seems to do what I need with some file tests even if it feels hacky (only using Perl for getting the file/directory so I don't have to sed/awk, shelling out for almost everything else).

#!/usr/bin/perl

use File::Basename;
use File::Copy;

$result = open (IN, "file.txt");
while (defined ($dataLine = <IN>)) {
        chomp ($dataLine);
        my $dir = dirname($dataLine);
        system("mkdir -p $dir");
        system("chown user:group $dir");
        my $oldPath = "/mnt/cifs" . $dataLine;
        copy($oldPath,$dataLine);
        system("chown user:group $dataLine");
        system("chmod 644 $dataLine");
}

Do you have the pax utility on CentOS6?

Do you really want all copied files to all be owned by a single user and group, or do you want to copy the user and group settings from each of the source files to the corresponding destination files?

If pax is available, you're running as root, and you'd like to copy the owner, group, mode, and timestamp settings of the source files to your destination files, I would think that something like:

sed 's|^|/mnt/cifs|' < cleanerror.txt | pax -rw -p e -s '|^/mnt/cifs||' /.

should do what you want. (NOTE that I have not tested this command line on my system when running with root privileges. Without root privileges, I get warnings that I can't change the ownership of a file [even when the requested change is a no-op] when I try this on OS X.)

Don't have the pax utility.

Yeah, the destination directories and files are all owned by a single user and group (non-root service account for the application). The source is a CIFS-mounted network drive from the old server and could be mounted as the non-root service account's UID, but I'm okay with having the implicitness of chown'ing in the script because I'm only concerned with how the files end up rather than what permission state they're coming from.

I'm sorry to hear that you don't have pax.

I'm extremely surprised that the bash script above is working for you. Here is your script with a few comments added:

#!/bin/bash

file='1.2699.gz' # This line does not affect the output produced by this script.
files=`cat cleanerror.txt`
filesdir=`sed -e 's/$file//g' cleanerror.txt
# Since $file is not expanded when it appears between single quotes, unless
# the string '$file' appears in some of the pathnames in cleanerror.txt, this
# line could be replaced by: filesdir="$files"
# So, both $files and $filesdir expand to your list of 5000 target files.

for i in $filesdir; do
mkdir -p $i # This will create a new directory for every target file,
done

for i in $files; do
cp "/mnt/cifs/$i" $i # This will add a new directory into the target file pathname.
chown user:group $i
# This above line will change the owner of the directory containing the
# target file; not the target file itself.
done

So, as an example, the processing of the line in cleanerror.txt:

/folder/path/path2/pathX/file.lua,d/1.2699.gz

will create a directory named:

/folder/path/path2/pathX/file.lua,d/1.2699.gz

and then copy the file:

/mnt/cifs/folder/path/path2/pathX/file.lua,d/1.2699.gz

to:

/folder/path/path2/pathX/file.lua,d/1.2699.gz/i.2699.gz

If you are running this script as the person whose userID is the one that you want to be the owner of the files, cp will do that for you without invoking chown. (If you need it, feel free to invoke chown on the files you create and on all of the directories you create [not just the leaf directories]. Note that invoking chown 5000 times will add somewhere between several seconds and a few minutes to the run the of your script. Figuring out which directories need to be created and have their ownership changed will also make the script more complex and slower.)

If you're trying to do what I think you're trying to do, you could try something like:

#!/bin/bash
while IFS='' read -r file
do      filedir="${file%/*}"
        if [ ! -d "$filedir" ]
        then    mkdir -p "$filedir"
        fi
        cp "/mnt/cifs/$file" "$file"
done < cleanerror.txt

That's easily remedied:

# yum install pax