Hi,
I do a weekly complete backup of my UNIFI controller (running on RPI4) and the files are placed in a folder.
I have setup a cronjob (as "su") to transfer the files, but I can't get the script to work, it executes fine.
I have set the backup folder on my RPI4 to CHMOD 777 (recursively)
I know that 777 is a bit much, but just to be sure that it isn't some permission error that is causing the problems I chose777.
At first I tried this:
#!/bin/bash
# usage: This script copies backup of Unifi to <snip> Ftp
IP_address="<snip>"
username="<snip>"
password="<snip>"
cd /var/lib/unifi/backup/autobackup
ftp -n > /home/pi/cronjobs/logs/UI_ftp_$$.log <<EOF
verbose
open $IP_address
quote USER $username
quote PASS $password
cd Backups/UI
mput *
bye
EOF
But this only copies 2 of the max 7 files in my backup folder.
(I have a 7 week retention on backups)
This is what is in the ftp-log from the crontab execution:
Verbose mode on.
Connected to <snip>
220 NASFTPD Turbo station 1.3.5a Server (ProFTPD)
331 Password required for <snip>
230 User <snip> logged in
250 CWD command successful
mput autobackup_6.1.71_20210418_0000_1618704000093.unf? 200 PORT command successful
150 Opening ASCII mode data connection for autobackup_6.1.71_20210418_0000_1618704000093.unf
226 Transfer complete
7381002 bytes sent in 0.25 secs (28.4114 MB/s)
mput autobackup_6.1.71_20210425_0200_1619316000018.unf? mput autobackup_6.1.71_20210502_0200_1619920800011.unf? mput autobackup_meta.json? mput ftp.txt? 221 Goodbye.
Then I tried to compile a txt-file with the Ftp-commands using individual PUT for each file, and then execute the file:
#!/bin/bash
HOST='<snip>'
USER='<snip>'
PASSWD='<snip>'
TYPE='.unf'
cd /var/lib/unifi/backup/autobackup
echo verbose >> ftp.txt
echo open $HOST >> ftp.txt
echo USER $USER >> ftp.txt
echo PASS $PASSWD >> ftp.txt
echo CD /Backups/UI >> ftp.txt
#echo ascii >> ftp.txt
for filename in *$TYPE; do
echo "put $filename"
done >> ftp.txt
echo bye >> ftp.txt
ftp -n > /home/pi/cronjobs/logs/ftp_$$.log <<EOF
ftp -in < ftp.txt
EOF
rm ftp.txt
The file comes out looking like this:
verbose
open <snip>
USER <snip>
PASS <snip>
CD /Backups/UI
put autobackup_6.1.71_20210418_0000_1618704000093.unf
put autobackup_6.1.71_20210425_0200_1619316000018.unf
put autobackup_6.1.71_20210502_0200_1619920800011.unf
bye
But that's even worse, now the log is allmost empty it just says:
?Invalid Command
As you may have noticed I run my Ftp on a QNAP NAS server and it works fine for other Ftp jobs and misc. tasks. I have set the user root to be my "Public" folder and that's why I do a "cd" to get the files into the right folder. I could set the root to the backup folder but that would cause problems for me with other jobs running Backups on the same RPI (e.g. my SNMP monitoring)
Yes, I could do that - but to my understanding it syncs both ways?
(I don't know that much about rsync)
I want to have a longer retention on my Ftp so two-way sync wouldn't do the job.
No, that is not how rsync works. You can easily transfer files one way or the other.
Not sure where you would come up with such an idea, to be honest.
Hmm. It's pretty basic that everyone should learn rsync working on unix/linux based systems.
No one I know uses ``ftp```. It's not secure. We use sftp, rsync, etc. all of which run over ssh.
See:
See Also:
Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.
Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file's data does not need to be updated.
Some of the additional features of rsync are:
o support for copying links, devices, owners, groups, and permissions
o exclude and exclude-from options similar to GNU tar
o a CVS exclude mode for ignoring the same files that CVS would ignore
o can use any transparent remote shell, including ssh or rsh
o does not require super-user privileges
o pipelining of file transfers to minimize latency costs
o support for anonymous or authenticated rsync daemons (ideal for mirroring)
FTP was not built to be secure. It is generally considered to be an insecure protocol because it relies on clear-text usernames and passwords for authentication and does not use encryption. Data sent via FTP is vulnerable to sniffing, spoofing, and brute force attacks, among other basic attack methods.
I have many servers on the net and most run daily backups. For each one I use an entry in a crontab file and a short rsync script to keep the backups "in sync" which means moving the new backups to the server where I keep all backups.
Not trying to sound too blunt, but this is a very basic system admin task we all do, BTW. I would ever use ftp for a task like this; and honestly have not used ftp in perhaps two decades or more, maybe three decades... but I cannot recall because it's not good to use ftp and like I said, I do not know of anyone who would use such an insecure protocol for file transfers.
Do not use ftp..... use rsync or sftp, but for backups synced with cron, I always use rsync..... it's made for this purpose.
Thank you for the thorough answer, but my original question was not regarding wich method to use, but why the method I have chosen didn't work.
I will consider rsync, but I still would like to know why MPUT doesn't work the way I expect it to do.
Since I do not use ftp for anything, and as I mentioned, no one I know uses ftp (because of the security issues and there are better tools to use), you might find it hard to get help; since few people actually use ftp
I'd echo everything that Neo has said here - ideally you want to be using a more modern solution that more easily supports what you're trying to do. However, I think the actual problem here is fairly simple: you need to add the -i flag to your ftp command to turn off mput confirmation prompts.
In my own testing using your initial script I would only ever get the first file uploaded, because ftp was prompting the user for confirmation for every file to mput - and of course it was never able to get it, since it was not attached to a terminal. When I changed the command to ftp -i -n in the script, it then ran fine, and all the expected files were uploaded.
For the sake of completeness, here's the exact script I used, and which was able to upload all my test files successfully:
#!/bin/bash
# usage: This script copies backup of Unifi to <snip> Ftp
IP_address="127.0.0.1"
username="username"
password="password"
cd /home/unixforum/382644/stuff_to_upload/
ftp -i -n > /home/unixforum/382644/ftp.log <<EOF
verbose
open $IP_address
quote USER $username
quote PASS $password
cd Backups/UI
mput *
bye
EOF
This was to and from a test server running CentOS 7 with vsftpd as the FTP server software and ftp as the client, so your mileage may vary, as they say, when connecting to other systems. But certainly in my own testing, the above script worked, and resulted in all the files being uploaded.
Hope this helps ! But ideally yes, do look at something like scp or rsync, since this would make your life far easier going forward.
Thanks to all of you.
I've switched to rsync and it works fine, I had to change some settings in my NAS but nothing major.
As far as the original Ftp problem goes...
The "-i" flag solved the problem and MPUT now transfers all the files.
I'll continue to use the rsync setup, but it's nice to know the solution to a problem no matter if I'm going to use or not
