I am trying to move alot (13,000 +) files from a Centos7 system to a Western Digital NAS. Here is my command:
rsync --verbose --log-file=/home/asterisk/Backup/rsynclog.txt --stats --progress -zvr /var/spool/asterisk/monitor/ sshd@xxx.xxx.xxx.xxx:/shares/Recordings
The file structure is such that there is a directory for each month 01-12 and a directory under that for each day 01 -31. there could be several hundred files in each day. When I run the command it transfers about 75 files and then exits. It exits without any info about what it did. I cannot find anythig that would limit the number of files transferred per session. I have this same setup working at two other sites without any issues. Any ideas on what is going on?
Thanks
@edlentz , hi, could be any number of potentials ....
- have you captured the logging from the job ?
- what is/was the exit status?
- is the nas full
- disk/h-ware errors/issues
- was the session dropped, timed-out
- bandwidth limit on the link/account/....
- system resources (memory exhausted)
- permission errors on some files (source +/ destination)
- ....
perhaps add some additional flags to the xfer
--dry-run # test the command without actually doing anything
--partial # will (re)start at where it left off
--avz # operate in verbose, archive mode, compressing for the transfer , is
decompressed once transfer is completed
EXAMPLE: rsync -avz foo:src/bar /data/tmp
from the manpage ....
This would recursively transfer all files from the directory
src/bar on the machine foo into the /data/tmp/bar directory on
the local machine. The files are transferred in archive mode,
which ensures that symbolic links, devices, attributes,
permissions, ownerships, etc. are preserved in the transfer.
Additionally, compression will be used to reduce the size of data
portions of the transfer.
Hi munkeHoller
Thanks for the reply.
I have a log file and I have this error:
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(638) [sender=3.1.2]
No exit status , running it from the CLI it reverts to the command prompt.
New NAS has over 3 TB available
No disk/hardware issues known . Ran dmesg | grep -i 'out of memory'
Got no entries just back to command prompt.
Running this from the source over a ssh session and my session didn't drop out.
Permissions are the same for all files/directories.
I will try the dry run to see if it fails and also will add the -partial which should help. In your example shouldn't your option be avzr instead of avz to recursively work through the directories?
Thanks again
For what it is worth here is the results from a dry run
2024/05/22 14:05:25 [27534] rsync[13181] (server generator) heap statistics:
2024/05/22 14:05:25 [27534] arena: 688128 (bytes from sbrk)
2024/05/22 14:05:25 [27534] ordblks: 3 (chunks not in use)
2024/05/22 14:05:25 [27534] smblks: 1
2024/05/22 14:05:25 [27534] hblks: 2 (chunks from mmap)
2024/05/22 14:05:25 [27534] hblkhd: 327680 (bytes from mmap)
2024/05/22 14:05:25 [27534] allmem: 1015808 (bytes from sbrk + mmap)
2024/05/22 14:05:25 [27534] usmblks: 0
2024/05/22 14:05:25 [27534] fsmblks: 64
2024/05/22 14:05:25 [27534] uordblks: 137408 (bytes used)
2024/05/22 14:05:25 [27534] fordblks: 550720 (bytes free)
2024/05/22 14:05:25 [27534] keepcost: 423752 (bytes in releasable chunk)
2024/05/22 14:05:25 [27534] Number of files: 10,323 (reg: 10,284, dir: 39)
2024/05/22 14:05:25 [27534] Number of created files: 10,315 (reg: 10,284, dir: 31)
2024/05/22 14:05:25 [27534] Number of deleted files: 0
2024/05/22 14:05:25 [27534] Number of regular files transferred: 10,284
2024/05/22 14:05:25 [27534] Total file size: 10,229,631,504 bytes
2024/05/22 14:05:25 [27534] Total transferred file size: 10,229,631,504 bytes
2024/05/22 14:05:25 [27534] Literal data: 0 bytes
2024/05/22 14:05:25 [27534] Matched data: 0 bytes
2024/05/22 14:05:25 [27534] File list size: 65,486
2024/05/22 14:05:25 [27534] File list generation time: 0.001 seconds
2024/05/22 14:05:25 [27534] File list transfer time: 0.000 seconds
2024/05/22 14:05:25 [27534] Total bytes sent: 728,263
2024/05/22 14:05:25 [27534] Total bytes received: 31,178
2024/05/22 14:05:25 [27534] sent 728,263 bytes received 31,178 bytes 138,080.18 bytes/sec
2024/05/22 14:05:25 [27534] total size is 10,229,631,504 speedup is 13,469.95 (DRY RUN)
i think the -avz will traverse all directories., --dry-run to test if unsure.
in addition to dmesg, you might want to look in other logs /var/log/syslog /var/log/messages etc for errors - not just out of memory
how is this job instantiated ? cron (show if so ...)
those signals point towards manual intervention or session ending ....
tks
I'll try a dry run with AVZ
Haven't looked in any other log files, will now
It is run within a bash script using expect to send the password for the NAS. and the script is run from a cron job. For now I am running the bash script from the CLI.
Yeah those signals are puzzling since I am the only one working on this. Unless there is something with the NAS. I have two others, same models working fine tho.
tks
You re right avz does run recursively through the directories.
you may want to add (experiment with) the --info= option
open this link and search for --info=progress , read carefully 
https://download.samba.org/pub/rsync/rsync.1