Transfer files from one server with bash script

Shell Programming and Scripting
1

Hello to all,

I want to copy from one server to another files of last 24 hours with size between 500MB and 2GB. The code below searches last files in 24 hours.

find . -mtime -1

In order to copy faster I'd like to compress the files before copying them.

How to automate the process of find the last files, then compress, copy them and finally erase the compressed files.

Maybe some could help me out.

Best regards

2

What OS and shell are you running?

3

Hello Jim,

Both origin and destination server when I send echo $0 the answer is bash and when I send uname -a the answer is x86_64 x86_64 GNU/Linux .

Best regards

4

Perhaps you can use rsync as the recipient of find .

-a  # preserve permissions and archive
-z  # will compress for transmission
-v  # is verbose
--dry-run  # is for testing, it will show the result without doing anything. Remove it to do the real work
--remove-source-files # will delete the files after the transfer is successful
--files-from=- # will pickup the list of files to transfer from stdin.
-0 # to handle find output
-e ssh # to transfer encrypted, via the ssh protocol

find /path/ -mtime -1 -type f -printf %P\\0 | rsync -azv --dry-run --remove-source-files --files-from=- -0  /path/ -e ssh user@hostname:/destination/

If you set your ssh with private/public key this can be automatic.

5

Hi Aia,

Sorry my lateness in response. I had issues with connection.

Source Server: is the server where the files I want to copy reside
Destination Server: is the server where I try to send the command to transfer the files from Source Server to Destination Server.

So, I'm trying to send the command you suggest in the way below in Destination Server but I get the errors below:. What I'm doing wrong?

Thanks for the help.

 find /path/ -mtime -1 -type f -printf %P\\0 | rsync -azv --dry-run --remove-source-files --files-from=- -0  /path/ -e ssh root@192.168.100.20:/destination/
 
 find: �/path/': No such file or directory
root@1192.168.100.20's password:
building file list ...
rsync: change_dir "/path/" failed: No such file or directory (2)
rsync error: errors selecting input/output files, dirs (code 3) at flist.c(2118) [sender=3.1.1]
6

/path/ was intended as an example; you have to substitute it for your own directory, the one you want to search from. Same goes with /destination/ , you have to substitute for the destination directory you want.

7

Hi Aia,

I don't know what I'm doing wrong.

The files I want to transfer/copy from are in Server A (Source Server) and it has the addres 192.168.100.20
The Server B would be the destination server

In Server A, the files I want to copy are in path "/SourceFiles"
In Server B I have the destination folder "/DestinationServerB"

I'm trying to send the command from the destination Server (Server B) in the way below but I still get the error.

find /SourceFiles/ -mtime -1 -type f -printf %P\\0 | rsync -azv --dry-run --remove-source-files --files-from=- -0  /SourceFiles/ -e ssh root@192.168.100.20:/DestinationServerB/

This command should be sent from Destination Server or from Source Server? I'd like to send the extraction script from the destination server.

Thanks again for the help

8

Hi Ophiuchus,

The assumption is that you are issuing the command from server A. Therefore the ip at root@ must be of server B and you are entering the ip of server A.

It is best if you always post the errors and warnings you are getting, even when it looks unimportant or does not make sense to you.

9

Hi Aia,

Below the error.

But I'd like to connect auntomatically via script to Server A (Source Server) from Server B (Destination Server) and extract the desired files. I want to avoid to set scripts in Source Server.

Is there a way to do this?

Source Server data:
user: root
password: xyzw
IP: 192.168.100.20

find /SourceFiles/ -mtime -1 -type f -printf %P\\0 | rsync -azv --dry-run --remove-source-files --files-from=- -0  /SourceFiles/ -e ssh root@192.168.100.20:/DestinationServerB/

DestinationServerB
find: �/SourceFiles': No such file or directory
root@192.168.100.20's password:
building file list ...
rsync: change_dir "/SourceFiles" failed: No such file or directory (2)
rsync error: errors selecting input/output files, dirs (code 3) at flist.c(2118) [sender=3.1.1]

Thanks for the help.