Moving files from parent path to multiple child path using bash in efficient way

Shell Programming and Scripting
1

Hi All,

Can you please provide some pointers to move files from Base path to multiple paths in efficient way.Folder Structure is already created.

/Path/AdminUser/User1/1111/Reports/aaa.txt to /Path/User1/1111/Reports/aaa.txt
/Path/AdminUser/User1/2222/Reports/bbb.txt to /Path/User1/2222/Reports/bbb.txt
/Path/AdminUser/User2/3333/Reports/ccc.txt to /Path/User2/3333/Reports/ccc.txt
/Path/AdminUser/User2/4444/Reports/ddd.txt to /Path/User2/4444/Reports/ddd.txt

I don't want to move directory structure. I want to move files from that source path to destination path on daily basis. I would need pointer to write a bash script to make sure aaa.txt would exactly go and sit only from /Path/AdminUser/User1/1111/Reports/ to /Path/User1/1111/Reports/ ........

One of geek folk helped provided with inputs to start as below, It still threw error, can we also write it in an efficient way?

#!/bin/bash
dir1=/Path
for i in "$dir1"/AdminUser/*; do
if [[ -d $i && ! -L $i ]]; then 
dir2="${i##*/}"
for j in "$i"/*; do 
if [[ -d $j && ! -L $j ]]; then 
j="${j##*/}"
mv "$i"/"$j"/Reports "$dir1"/"$dir2"/"$j"/
fi
done
fi
done
2

Your requirement isn't clear.

  1. Does the file hierarchy you're modifying contain symbolic links matched by the patterns /Path/AdminUser/* and /Path/AdminUser/*/* ? Your textual description didn't say anything about symbolic links, but your (non-working) code explicitly ignores them. If symbolic links are found, what do want to have done with them?
  2. Are you trying to move all files in the file hierarchy rooted in /Path/AdminUser/ up one level in that hierarchy to /Path/ ; or are you just trying to move regular files with pathnames matching the pattern /Path/AdminUser/*/*/Reports/*.txt to the corresponding pathname after removing AdminUser/ from the source pathname?
  3. If a directory in the target pathname does not already exist, should your script report an error and move on, should it create the missing directories, or should it silently ignore that source pathname?
  4. Do you want to leave the (empty) source directories in place after files are moved out of them, or should emptied directories be removed?
  5. Is the entire file hierarchy rooted in /Path in a single filesystem?

Instead of saying "It still threw error, can we also write it in an efficient way?", show us exactly what "error(s) it produces" (in CODE tags).

What do you believe it is doing inefficiently?

3

Hi Cragun,

Please see answers for questions, which helps provide complete picture:

  1. We dont have symbolic links in path, so we can ignore it.

  2. I am trying to move regular files with pathnames matching the pattern /Path/AdminUser/*/*/Reports/*.txt to the corresponding pathname after removing AdminUser/ from the source pathname.

  3. It can ignore that source pathname.

  4. Empty source directories should be in place after files are moved out of them.

  5. Yes the entire file hierarchy rooted in /Path in a single filesystem.

To improve performance can any thing like xargs play a major role

4

You man have some more success using rsync perhaps like this:

$ rsync -a --remove-source-files /Path/AdminUser/ /Path
5

@ Chubler
Thank you. rsync works, but since in my scenario, its not just folder structure, I have to make sure security & ownership of file system isn't affected so I was inclined to use script to move files in loop from parent path to child path, with out touching directories.

6

You have said that a "geek" gave you a non-working script that uses bash , but you haven't said what shell(s) are available for you to use nor what operating system(s) are present on systems that will be running this script.

The following script will work with any POSIX-conforming shell on any system that has a shell that conforms to the 1992 or any later version of the POSIX Shell and Utilities standard (and many shells that support a fairly common subset of POSIX requirements) such as any version of ksh or bash :

#!/bin/ksh
# This script works with any POSIX conforming shell.
dir1="/Path"
cd "$dir1"
for srcd in AdminUser/*/*/Reports
do	targetd=${srcd#*/}
	if [ ! -d "$srcd" ] || [ ! -d "$targetd" ]
	then	continue	# Source or target directory does not exist.
	fi
	cd "$srcd" > /dev/null
	for file in *.txt
	do	if [ "$file" = '*.txt' ]
		then	break	# No *.txt files to move in this directory
		else	echo mv *.txt "$dir1/$targetd"	# Move the files
			break
		fi
	done
	cd - > /dev/null
done

If you have a recent ksh that expands ~(N)*.txt to an empty list if there are no files in the current directory ending with .txt or if you have a recent bash that expands *.txt to an empty list after the command shopt -s nullglob , then the 2nd for loop in the above script can be simplified as in:

#!/bin/ksh
# This script works with a recent Korn shell that recognizes: ~(N)*.txt
# to expand to nothing if there are no files in the current directory ending
# with ".txt"
dir1="/Path"
cd "$dir1"
for srcd in AdminUser/*/*/Reports
do	targetd=${srcd#*/}
	if [ ! -d "$srcd" ] || [ ! -d "$targetd" ]
	then	continue	# Source or target directory does not exist.
	fi
	cd "$srcd" > /dev/null
	for file in ~(N)*.txt
	do	echo mv *.txt "$dir1/$targetd"	# Move the files
		break
	done
	cd - > /dev/null
done

or:

#!/bin/bash
# This script works with a recent bash shell that recognizes: shopt -s nullglob
# followed by: *.txt
# to expand to nothing if there are no files in the current directory ending
# with ".txt"
shopt -s nullglob
dir1="/Path"
cd "$dir1"
for srcd in AdminUser/*/*/Reports
do	targetd=${srcd#*/}
	if [ ! -d "$srcd" ] || [ ! -d "$targetd" ]
	then	continue	# Source or target directory does not exist.
	fi
	cd "$srcd" > /dev/null
	for file in *.txt
	do	echo mv *.txt "$dir1/$targetd"	# Move the files
		break
	done
	cd - > /dev/null
done

If the mv commands echoed by the above scripts do what you want, remove the echo shown in red to actually execute the mv commands.

All of these scripts assume that (since they move into the various source directories before expanding the list of files to be moved) that list won't exceed ARG_MAX limits (so xargs or similar utilities to combine groups of files into single invocations of mv ) won't be needed. They all produce one mv command for each source directory that contains files to be moved that also have an existing target directory to receive those files. If some directories have a huge number of files to be moved with names long enough to exceed ARG_MAX, the 2nd for loop can be modified to feed the list of files to be moved to xargs , but it will run a little bit slower (and depending on what operating system you're using), may have problems with filenames containing whitespace characters and may also require special processing to specify the target directory in the appropriate place in an xargs -generated mv command.

1 Like
7

Thanks a lot Don for detail explanation and scripts for each scenario. It really helps !!!