Copy files in respective directories

Shell Programming and Scripting
1

Hi Guys,

I need to copy the files to respective directories based on name of the file. My script is something like below

con=$1

for file in `cat $con` 
do
file_tmp=$(ls -t1 $path|  grep -i $file | head -n 1)
echo $file_tmp

if [ -f $path$file_tmp ]
then
	cp $path$file_tmp $DIR/ap

	if [ "$?" = "0" ]
	then
		echo "file copied"
				 
               
		
	else
		echo "Failed copy"
			
	fi
else 
	echo "$file_tmp not present "

fi
done

con.txt

apple
mango
fish

if its apple file then copy to $DIR/ap
if its mango file then copy to $DIR/man
if its fish file then copy to $DIR/fis

I need to copy the file in respctive paths by reading the file naming convention from config file

2

How to determine the target directory? There doesn't seem to be a rule behind it. Can con.txt be extended to contain it? What be the "file naming convention"?
Your $path is undefined, as is your $DIR .

3

Hi RudiC,

I also though the same way to determine to add in con.txt respective path. for eg. or a better way to achieve

apple,/dir/ap
mango,/dir/man
fish,/dir/fis

path=input_files/path/
dir=/dir/
4

Hello Master_Mind,

Could you please try following and let me know if this helps you. Considering that your paths dir/app, dir/man all are already created.

DIR=/my/directory/path
awk -vdir="$DIR"  'function copy(path){system("cp " $0 "  " path)} {if($0 == "apple"){copy(dir"/ap")};if($0 == "fish"){copy(dir"/fis")};if($0 == "mango"){copy(dir"/man")}}'    Input_file

EDIT: Adding a non-one liner form of solution too now.

DIR=/my/directory/path
awk -vdir="$DIR"  'function copy(path){
                                        system("cp " $0 "  " path)
                                      }
                                      {
                                        if($0 == "apple"){
                                                                copy(dir"/ap")
                                                         };
                                        if($0 == "fish") {
                                                                copy(dir"/fis")
                                                         };
                                        if($0 == "mango"){
                                                                copy(dir"/man")
                                                         }
                                      }
                  '    Input_file

Thanks,
R. Singh

1 Like
5

How about

path=/path/to
while IFS="," read FN PTH; do echo cp $path/$FN $PTH/$FN; done <con.txt
cp /path/to/apple /dir/ap/apple
cp /path/to/mango /dir/man/mango
cp /path/to/fish /dir/fis/fish

Remove echo when happy with proposals.

And, try to be consistent with trailing slashes when defining paths.

6

Thanks Ravi,

But i would require to print messages like how i need if successfully copied or not and if the file doesnt present in input directory to print not present

7

Hello Master_Mind,

Could you please try following and let me know if this helps you.

cat copy_files.ksh
DIR=/tmp
awk -vdir="$DIR"  'function copy(path){
                                        system("cp " $0 "  " path";if [[ $? == 0 ]]; then echo " $0 " file copied successfully to " path ".; else echo Please check there is an issue while copying file.; fi")
                                      }
                                      {
                                        if($0 == "apple"){
                                                                copy(dir"/ap")
                                                         };
                                        if($0 == "fish") {
                                                                copy(dir"/fis")
                                                         };
                                        if($0 == "mango"){
                                                                copy(dir"/man")
                                                         }
                                      }
                  '    Input_file

EDIT: If you want to get that a file is NOT present error then following may help you in same too.

cat copy_files.ksh
DIR=/tmp
awk -vdir="$DIR"  'function copy(path){
                                        system("if [[ -f " $0 " ]]; then cp " $0 "  " path";if [[ $? == 0 ]]; then echo " $0 " file copied successfully to " path ".; else echo Please check there is an issue while copying file.; fi; else echo " $0 " file not present.; fi")
                                      }
                                      {
                                        if($0 == "apple"){
                                                                copy(dir"/ap")
                                                         };
                                        if($0 == "fish") {
                                                                copy(dir"/fis")
                                                         };
                                        if($0 == "mango"){
                                                                copy(dir"/man")
                                                         }
                                      }
                  '    Input_file

Thanks,
R. Singh

1 Like
8

Thanks Ravi once again, moreover i am taking the latest file if you check in my script. I take the latest file and copy that into respective path. Sorry i missed one more thing to say. If my file name is fish then i need to create a directory

cur_date=$(date +%Y%m%d)
mkdir $DIR/fis/$cur_date
cp /path/to/fish /dir/fis/$cur_date/
if not found then copy from previous directory
cp /dir/fis/$cur_date-1/ /dir/fis/$cur_date/
$(ls -t1 $path|  grep -i $file | head -n 1)
9

Hello Master_Mind,

Could you please try this and let me know if this helps you.

cat copy_files.ksh
DIR="/tmp"
awk -vdir="$DIR"  'function copy(path){
                                        system("if [[ -f " $0 " ]]; then if [[ ! -d  " path " ]]; then mkdir "path"; echo " path " directory created successfully now.; fi; cp " $0  FS path  ";if [[ $? == 0 ]]; then echo " $0 " file copied successfully to " path ".; else echo Please check there is an issue while copying file.; fi; else echo " $0 " file not present.; fi")
                                      }
                                      {
                                        if($0 == "apple"){
                                                                copy(dir"/ap")
                                                         };
                                        if($0 == "fish") {
                                                                copy(dir"/fis")
                                                         };
                                        if($0 == "mango"){
                                                                copy(dir"/man")
                                                         }
                                      }
                  '    Input_file

EDIT: Adding a non-one liner form of system 's command above for better understanding.

cat copy_files.ksh
DIR="/tmp"
awk -vdir="$DIR"  'function copy(path){
                                        system("if [[ -f " $0 " ]]; \
                                        then if [[ ! -d  " path " ]]; \
                                        then mkdir "path"; echo " path " \
                                        directory created successfully now.; \
                                        fi; cp " $0  FS path  ";if [[ $? == 0 ]]; \
                                        then echo " $0 " file copied successfully to " \
                                        path ".; else echo Please check there is an issue while copying file.; \
                                        fi; else echo " $0 " file not present.; fi")
                                      }
                                      {
                                        if($0 == "apple"){
                                                                copy(dir"/ap")
                                                         };
                                        if($0 == "fish") {
                                                                copy(dir"/fis")
                                                         };
                                        if($0 == "mango"){
                                                                copy(dir"/man")
                                                         }
                                      }
                  '    Input_file

Thanks,
R. Singh

1 Like
10

Thanks Ravi,

But creating the system date directory is only for file name which has fish not for other files. Only for fish file i would require to mkdir with cur_date and copy the file if the file is not found then copy file from previous month cur_date.

for eg

mkdir dir/fish/2016-03-01/
cp from input directory to dir/fish/2016-03-01/

if file not found in input directory then copy from
dir/fish/2016-02-01/ to dir/fish/2016-03-01/

11

Hello Master_Mind,

Not clear to me, could you please put all your requirements all together and give more useful samples to make it clear to us.

Thanks,
R. Singh

1 Like
12

Sure Ravi, let me explain you clear sorry for confusing.

config file

apple
mango
fish

input file directory which is comman directory

dir/input/
apple.txt
mango.txt
mango_1.txt
fish.txt

copying method

cp dir/input/apple.txt output/ap/
cp dir/input/mango_1.txt output/man/
cp dir/input/fish.txt output/fish/2017-03-21/

above mango_1 is being copied since it is the latest file placed in input directory
for fish.txt - since its present in input directory i need to first make directory with current date say 2017-03-21 and copy the file
if suppose fish.txt is not present in input direc then copy from previous month directory considering 2017-02 YYYY-MM because date can be anything

cp output/fish/2017-02*/  output/fish/2017-03-21/

all the files should be taken latest file from input directory thats why i would have added this command

ls -t1 $path|  grep -i $file | head -n 1
13

Hello Master_Mind,

Could you please try following and let me know if this helps you.

cat copy_files.ksh
DIR="/tmp"
INPUT=/edmdidx03/admin/a553454
DATE=$(date +%Y-%m-%d)
DATE_PRE=$(date +%Y-%m-%d -d"1 month ago")
awk -vdate=$DATE -vdate_pre=$DATE_PRE -vdir="$DIR"  -vexcept="fis" -vinput=$INPUT 'function copy(path){
                                        system("set -x;if [[ -f " input"/"$0 " ]] ; then  if [[ ! -d " path " ]] ; then  mkdir " path " echo " path " got created successfully. ; fi ; if [[ " $0 " == " except " ]] ; then  mkdir " path"/"date " ; cp " $0 FS path"/"date " ; else  cp " $0 FS path  " ; fi ; else  if [[ ! -d " dir"/"except"/"date_pre" ]] ; then  echo seems directory named " dir"/"except"/"date_pre "not existing. ; fi ; echo test; if [[ " $0 " == " except " ]] ; then  echo chumma ;cp " dir"/"except"/"date_pre"/*" FS dir"/"except"/"date" ; fi ; fi")
                                      }
                                      {
                                        if($0 == "apple"){
                                                                copy(dir"/ap")
                                                         };
                                        if($0 == "fis") {
                                                                copy(dir"/fis")
                                                         };
                                        if($0 == "mango"){
                                                                copy(dir"/man")
                                                         }
                                      }
                  '     Input_file

EDIT: Adding better readable form of above code too now.

cat copy_files.ksh
DIR="/tmp"
INPUT=/edmdidx03/admin/a553454
DATE=$(date +%Y-%m-%d)
DATE_PRE=$(date +%Y-%m-%d -d"1 month ago")
awk -vdate=$DATE -vdate_pre=$DATE_PRE -vdir="$DIR"  -vexcept="fis" -vinput=$INPUT 'function copy(path){
                                        system("set -x;if [[ -f " input"/"$0 " ]] ; \
                                        then  if [[ ! -d " path " ]] ; then  mkdir " \
                                        path " echo " path " got created successfully. ; \
                                        fi ; if [[ " $0 " == " except " ]] ; then  mkdir " \
                                        path"/"date " ; cp " $0 FS path"/"date " ; else  cp " \
                                        $0 FS path  " ; fi ; else  if [[ ! -d " dir"/"except"/"date_pre" ]] ; \
                                        then  echo seems directory named " dir"/"except"/"date_pre "not existing. ; \
                                        fi ; echo test; if [[ " $0 " == " except " ]] ; then  echo chumma ;cp " \
                                        dir"/"except"/"date_pre"/*" FS dir"/"except"/"date" ; fi ; fi")
                                      }
                                      {
                                        if($0 == "apple"){
                                                                copy(dir"/ap")
                                                         };
                                        if($0 == "fis") {
                                                                copy(dir"/fis")
                                                         };
                                        if($0 == "mango"){
                                                                copy(dir"/man")
                                                         }
                                      }
                  '    Input_file

PS: Some few points in above code, like variable named except is for the files which you want to check if they are not present then copy them from previous month's folder. Though I tested it but you need to check it completely for all the things.
I added code set -x into it for tracing purposes, you could remove it if you want to remove.

Thanks,
R. Singh

1 Like
14

Thanks Ravi for your help, One thing i wanted to know can you let me know why you care creating the directory for other files apart from fish, because i already have source directory created only for fish file i need to create directory with date folder

dir/fis/date

then  if [[ ! -d " path " ]] ; then  mkdir " \
                                        path " echo " path " got created successfully. ; \

And also I think I can your previous code that file successfully copied /not copied / file not present in input directory message in same function

15

Hello Master_Mind,

That was to make sure in case directory is not present it should create it else it will simply copy it. Let me know if you have any queries on same.

Thanks,
R. Singh

16

Thanks ravi and one more thing just wanted to add regarding the echo messages hope I can add in same function. Once again is there a way to achieve in doing in loops instead of going with awk.
And one more thing Ravi, my files in input direcortry has mango_1.txt,apple_1.txt. you code will check exact file name in config file which fail in copying. I need to take if the file name contains like apple* and take the latest file only to copy

I tried something like this

conf=$1

filepath=/input/
output=/output/
for file in `cat $confFile` 
do
file_temp=$(ls -t1 $filepath|  grep -i $file | head -n 1)
echo $file_temp

if [ "$file" == "apple" ]
then
if [ -f $filepath$file_temp ]
then
	echo "Copying $file_temp file to location"

      	cp $filepath$bt_file $output/ap/ 

	if [ "$?" = "0" ]
	then
		echo "Successfully copied"
	else
		echo "failed copying"
	fi
else 
	echo "No apple file in $filepath "

fi

elif [ $file == "mango" ]
then
if [ -f $filepath$file_temp ]
then
	echo "Copying $file_temp file to location"
      	cp $filepath$file_temp $output/man/ 

	if [ "$?" = "0" ]
	then
		echo "Successfully copied"
	
	else
		echo "failed copying"	
	fi
else 
echo "No mango file in $filepath "    
fi
fi

done
17

Hello Master_Mind,

As per your request following is in pure shell(uses GNU date too in it). Could you please try following and let me know if this helps you.

cat check_files.ksh
DIR=/tmp
SEARCH_DIR=`pwd`
DATE=$(date +%Y"/"%m"/"%d)
DATE_PREVIOUS_MONTH=$(date --date='-1 month' +%Y"/"%m"/"%d)
 
EXCEPTION_GET_FILE () {
                        for var in "$@"
                        do
                                exception_file=$var
                                if [[ -f $SEARCH_DIR"/"$exception_file ]]
                                then
                                        if [[ -d $DIR"/"$var"/"$DATE_PREVIOUS_MONTH ]]
                                        then
                                                mkdir $DIR"/"$var"/"$DATE
                                                cp -r $DIR"/"$var"/"$DATE_PREVIOUS_MONTH/*  $DIR"/"$var"/"$DATE
                                        else
                                                echo "Previous month directory " $DIR"/"$var"/"$DATE_PREVIOUS_MONTH " is NOT present for exception file named " $exception_file
                                                if [[ -d $DIR"/"$var ]]
                                                then
                                                        mkdir $DIR"/"$var"/"$DATE
                                                else
                                                        mkdir -p $DIR"/"$var"/"$DATE
                                                fi
                                                cp $exception_file $DIR"/"$var"/"$DATE
                                                CHECK_FILES_COPIED_STATUS $DIR $var"/"$DATE $exception_file
                                        fi
                                else
                                        echo "File named " $exception_file " itself NOT found in directory " $SEARCH_DIR
                                fi
                        done
                      }
CHECK_FILES_COPIED_STATUS () {
                                DIR=$1
                                dir=$2
                                REPLY=$REPLY
                                if [[ $? == 0 ]]
                                then
                                        echo "File named " $REPLY " has been successfully copied to directory named " $DIR"/"$dir " now."
                                else
                                        echo "please check seems there is some issue with file copy."
                                fi
                             }
GET_FILE () {
for var in "$@"
do
        input=${var%,*}
        dir=${var##*,}
if [[ -f $SEARCH_DIR"/"$input ]]
then
        while IFS=  read -r -d $'\0'
        do
                if [[ ! -d $DIR"/"$dir ]]
                then
                        mkdir $DIR"/"$dir
                        cp $REPLY $DIR"/"$dir
                        CHECK_FILES_COPIED_STATUS $DIR  $dir
                else
                        cp $REPLY  $DIR"/"$dir
                        CHECK_FILES_COPIED_STATUS $DIR  $dir
                fi
        ##echo $REPLY
        done < <(find $SEARCH_DIR -iname "${input}*" -print0)
else
        echo "File named " $input " NOT found in directory " $SEARCH_DIR
fi
done
            }
## Calling normal files to be copied here..
GET_FILE apple,app  mango,man
## Calling exception files to be checked and copied from last month's date to today's date....
EXCEPTION_GET_FILE fish

Above script's output will be as follows.

 ./check_files.ksh
File named  /singh_is_King/prince/R. Singh/apple_1.txt  has been successfully copied to directory named  /tmp/app  now.
File named  /singh_is_King/prince/R. Singh/apple  has been successfully copied to directory named  /tmp/app  now.
File named  /singh_is_King/prince/R. Singh/apple12111  has been successfully copied to directory named  /tmp/app  now.
File named  /singh_is_King/prince/R. Singh/apple1.txt  has been successfully copied to directory named  /tmp/app  now.
File named  /singh_is_King/prince/R. Singh/mango  has been successfully copied to directory named  /tmp/man  now.
File named  /singh_is_King/prince/R. Singh/path/mango  has been successfully copied to directory named  /tmp/man  now.
File named  /singh_is_King/prince/R. Singh/mango1211.txt  has been successfully copied to directory named  /tmp/man  now.
File named  /singh_is_King/prince/R. Singh/mango1211  has been successfully copied to directory named  /tmp/man  now.
File named  fish  itself NOT found in directory  /singh_is_King/prince/R. Singh

So above code will look for all the files with similar names like mango, mango1211.txt, mango1211 etc. Also there are 2 functions in above script named GET_FILE and EXCEPTION_GET_FILE so as per name suggests itself function named EXCEPTION_GET_FILE for files like fish etc. So here is how you need to run them, we need to pass arguments like EXCEPTION_GET_FILE exception_file_name and GET_FILE file_name,it's directory_name another_file_name,it's directory_name and so on. Also for exception files I am not checking similar names, I am exactly checking the same name as exception file name only.

Kindly check it and let me know if this helps you.

NOTE: I placed few test files with similar names into my test directory for testing purposes. Also exception function doesn't have argument of it's directory name so if you need it you could take a look to normal function one and could try to take same method into exception function too.

Thanks,
R. Singh

1 Like