Continuous Copying from a directory to another.

Shell Programming and Scripting
1

Hello Folks,

Looking for a script, where i can check for the existing of the files in a directory and copying to another.
How can i achieve it in loop for over period of time.

whatever files comes into the folder copied in another without duplicate and should be continuous loop.

Regards,
Sadique

2

Dear sadique.manzar,

I have a few to questions pose in response first:-

  • Is this homework/assignment? There are specific forums for these.
  • What have you tried so far?
  • What output/errors do you get?
  • What OS and version are you using?
  • What are your preferred tools? (C, shell, perl, awk, etc.)
  • What logical process have you considered? (to help steer us to follow what you are trying to achieve)

Most importantly, What have you tried so far?

There are probably many ways to achieve most tasks, so giving us an idea of your style and thoughts will help us guide you to an answer most suitable to you so you can adjust it to suit your needs in future.

We're all here to learn and getting the relevant information will help us all.

Kind regards,
Robin

3

hereis what i am trying:

#!/bin/sh
DATE=`date +"%d-%m-%Y %H:%M:%S"`
path=/xyz/S1
while true
do
	for  files in `printf "%s\n" $path/*.RTM 
	do
		if [[ -f $files ]] && [[ -s $files ]]
		then
			cp -n $files  /xxx
		else
			echo "file doesn't exist" >/dev/null
		fi	
	done
done
1 Like
4

The main things to consider are:

What about files that get modified? Do you want to copy them too? Or do files in this directory never get modified?

What about files that are still being written to/created? Do you want to copy these incomplete files regardless?

What lag in the copying can you tolerate? If a cron job ran every 5 minutes would that suffice? If you are trying to mirror in real-time then some mirroring suite would probably do the job better.

5

Dear Hicksd8,

First of all nice question.

  • What about files that get modified? Do you want to copy them too? Or do files in this directory never get modified?
    Never get modified, its like fly here and then from here to different server.
    .
  • What about files that are still being written to/created? Do you want to copy these incomplete files regardless?
    While incomplete the extension name is .temp, so my script is simply ruling them out.
    .
  • What lag in the copying can you tolerate? If a cron job ran every 5 minutes would that suffice? If you are trying to mirror in real-time then some mirroring suite would probably do the job better.
    As I said, files fly from this server to other. so running every 5 mins may miss files.

Regards,
Sadique

6

This could prove quite difficult if the length of time between the file being a .temp and being moved out is just a few seconds.

Do you have any access to the script(s) which is either creating the files or moving the files? It might be easier to rename the files from .temp to .temp2 and you copy all .temp2 files, then rename them suitable for the move script. Can you get into the overall chain of the process?

I'm scratching my head as to how you can guarantee catching all files if they only exist for a second or two. It would only take your script not to be scheduled (due to CPU load or whatever) for a few seconds in order to miss files going through.

7

Would iWatch be a possible help here? I think you can defined directories to watch and can take action when a file is updated.

The question is also 'What removes the files?' You may still miss files if you don't get to them before the delete happens. How much slack are we thinking? If this is (and I'm just guessing) an FTP server that people deliver to and there is another server that polls it to collect and remove files then there will be a race to get to the file, which is never a good position to be in.

Perhaps your process could be adjusted so that there is an in-bound directory for clients to create files and an out-bound for something to poll and remove the files (presumably after reading them) If you can separate these and have your code do the move after it has done the action you want, this might give you a working solution.

How time critical is the movement of files? If you are trading shares or processing payments then maybe your process could copy the file to the out-bound directory and clean up the in-bound file when it has done what extra you need.

Can you elaborate on what is going on?

Thanks, in advance,
Robin

1 Like
8

You are still ignoring the fact the a file might still be arriving as you copy it, with the result that the copy may not be the entire file.
sorry, I missed your comment about .temp in post 5.

Why not simply modify the process that renames the file to also do the copy.

9

The echo-line doesn't make any sense. You are writing something to the bitbucket? What for? Also, you create a variable DATE, but never use it. Finally, shell scripts which really loop permanently and do file operations continuously, might consume a considerable amount of processing power. You should at least sleep one or two seconds between iterations.

Having said this, rsync might be easier to use:

rsync --min-size=1 $path/*.RTM /xxx

This obviates the need for the nested for loop and the checking for the file size.

1 Like