I am using rsync
tool to send some files to usb storage device. The usb storage has NTFS filesystem.
sudo blkid /dev/sdd
/dev/sdd: LABEL="16gb" UUID="4B5DC2F908E16D3F" TYPE="ntfs" PTTYPE="dos"
cat /proc/mounts
dev/sdd /media/smc-desktop-07/16gb fuseblk rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other,blksize=4096 0 0
ls -lh /media/smc-desktop-07/
total 4.0K
drwxrwxrwx 1 smc-desktop-07 smc-desktop-07 4.0K Apr 11 15:13 16gb
The same error occurs for FAT32 filesystem.
def test_rsync(source, dest):
print("=" * 30)
st = time.perf_counter()
try:
send_folders = subprocess.run(
[
"rsync",
"-a", # rescursive, preserves owner, preserves permissions, preserves modification times, preserves group, copies symlinks as symlinks, preserves device files.
"-c", # https://askubuntu.com/questions/112863/rsync-not-working-between-ntfs-fat-and-ext ## takes a lot of time
"-H", # preserves hard-links
"-A", # preserves ACLs
"-X", # preserves extended attributes
"-E", # https://serverfault.com/questions/43014/copying-a-large-directory-tree-locally-cp-or-rsync -> for OSX
"-x", # don't cross file-system boundaries
"-vv", # increase verbosity
"--modify-window=2",
"--no-whole-file",
"--no-compress",
"--ignore-existing",
"--stats",
"--progress", # show progress during transfer
"--numeric-ids", # don't map uid/gid values by user/group name
"--itemize-changes",
"--chmod=a+rw", # FAT / FAT32 formatted drives don't support file permissions.
"--partial-dir=tmp-rsync-partial",
f"{source}",
f"{dest}",
]
)
except KeyboardInterrupt as e:
print("Keyboard Interrupted")
print(f"Time taken: {time.perf_counter()-st} sec")
print("Source size: {} GB".format(float(foldersize(source)) / 1e9))
print("Dest size: {} GB".format(float(foldersize(dest)) / 1e9))
print("=" * 30)
test_rsync(
"tahkom",
"/media/smc-desktop-07/16gb/smartcow-nx-devkit/",
)
Attaching partial output of copying a folder to usb
sending incremental file list
delta-transmission enabled
rsync: recv_generator: mkdir "/media/smc-desktop-07/16gb/smartcow-nx-devkit/tahkom/tahakom-approver/.git/branches" failed: Input/output error (5)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/media/smc-desktop-07/16gb/smartcow-nx-devkit/tahkom/tahakom-approver/.git/hooks" failed: Input/output error (5)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/media/smc-desktop-07/16gb/smartcow-nx-devkit/tahkom/tahakom-approver/.git/info" failed: Input/output error (5)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/media/smc-desktop-07/16gb/smartcow-nx-devkit/tahkom/tahakom-approver/.git/logs" failed: Input/output error (5)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/media/smc-desktop-07/16gb/smartcow-nx-devkit/tahkom/tahakom-approver/.git/objects" failed: Input/output error (5)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/media/smc-desktop-07/16gb/smartcow-nx-devkit/tahkom/tahakom-approver/.git/refs" failed: Input/output error (5)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/media/smc-desktop-07/16gb/smartcow-nx-devkit/tahkom/tahakom-approver/client-mui/node_modules" failed: Input/output error (5)
*** Skipping any contents from this failed directory ***
cd+++++++++ tahkom/
cd+++++++++ tahkom/tahakom-approver/
>f+++++++++ tahkom/tahakom-approver/vars-dev.env
124 100% 121.09kB/s 0:00:00 (xfr#10, ir-chk=1457/1470)
cd+++++++++ tahkom/tahakom-approver/.git/
>f+++++++++ tahkom/tahakom-approver/.git/HEAD
42 100% 41.02kB/s 0:00:00 (xfr#11, ir-chk=1449/1470)
>f+++++++++ tahkom/tahakom-approver/.git/config
313 100% 305.66kB/s 0:00:00 (xfr#12, ir-chk=1448/1470)
>f+++++++++ tahkom/tahakom-approver/.git/description
73 100% 71.29kB/s 0:00:00 (xfr#13, ir-chk=1447/1470)
>f+++++++++ tahkom/tahakom-approver/.git/index
29,801 100% 28.42MB/s 0:00:00 (xfr#14, ir-chk=1446/1470)
>f+++++++++ tahkom/tahakom-approver/.git/packed-refs
926 100% 904.30kB/s 0:00:00 (xfr#15, ir-chk=1445/1470)
cd+++++++++ tahkom/tahakom-approver/.git/branches/
cd+++++++++ tahkom/tahakom-approver/.git/hooks/
cd+++++++++ tahkom/tahakom-approver/.git/info/
cd+++++++++ tahkom/tahakom-approver/.git/logs/
cd+++++++++ tahkom/tahakom-approver/.git/objects/
cd+++++++++ tahkom/tahakom-approver/.git/refs/
cd+++++++++ tahkom/tahakom-approver/client-mui/
>f+++++++++ tahkom/tahakom-approver/client-mui/package-lock.json
962,394 100% 61.19MB/s 0:00:00 (xfr#20, ir-chk=1405/1470)
>f+++++++++ tahkom/tahakom-approver/client-mui/package.json
2,042 100% 132.94kB/s 0:00:00 (xfr#21, ir-chk=1404/1470)
rsync: mkstemp "/media/smc-desktop-07/16gb/smartcow-nx-devkit/tahkom/tahakom-approver/.dockerignore.xzCtNn" failed: Input/output error (5)
rsync: mkstemp "/media/smc-desktop-07/16gb/smartcow-nx-devkit/tahkom/tahakom-approver/.gitignore.ycaWjL" failed: Input/output error (5)
rsync: mkstemp "/media/smc-desktop-07/16gb/smartcow-nx-devkit/tahkom/tahakom-approver/.pylintrc.oe1BQ8" failed: Input/output error (5)
rsync: mkstemp "/media/smc-desktop-07/16gb/smartcow-nx-devkit/tahkom/tahakom-approver/.README.md.lTennw" failed: Input/output error (5)
cd+++++++++ tahkom/tahakom-approver/client-mui/node_modules/
rsync: mkstemp "/media/smc-desktop-07/16gb/smartcow-nx-devkit/tahkom/tahakom-approver/.git/.packed-refs.9RZUfH" failed: Input/output error (5)
rsync: mkstemp "/media/smc-desktop-07/16gb/smartcow-nx-devkit/tahkom/tahakom-approver/client-mui/.dockerignore.jO4AN4" failed: Input/output error (5)
rsync: recv_generator: mkdir "/media/smc-desktop-07/16gb/smartcow-nx-devkit/tahkom/tahakom-approver/client-mui/public" failed: Input/output error (5)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/media/smc-desktop-07/16gb/smartcow-nx-devkit/tahkom/tahakom-approver/client-mui/src" failed: Input/output error (5)
*** Skipping any contents from this failed directory ***
cd+++++++++ tahkom/tahakom-approver/client-mui/public/
cd+++++++++ tahkom/tahakom-approver/client-mui/src/
rsync: recv_generator: mkdir "/media/smc-desktop-07/16gb/smartcow-nx-devkit/tahkom/tahakom-approver/database/db_data" failed: Input/output error (5)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/media/smc-desktop-07/16gb/smartcow-nx-devkit/tahkom/tahakom-approver/migrations/__pycache__" failed: Input/output error (5)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/media/smc-desktop-07/16gb/smartcow-nx-devkit/tahkom/tahakom-approver/migrations/versions" failed: Input/output error (5)
*** Skipping any contents from this failed directory ***
rsync: recv_generator: mkdir "/media/smc-desktop-07/16gb/smartcow-nx-devkit/tahkom/tahakom-approver/prod-setup" failed: Input/output error (5)
total: matches=0 hash_hits=0 false_alarms=0 data=1022996
rsync[16162] (sender) heap statistics:
arena: 55398400 (bytes from sbrk)
ordblks: 365 (chunks not in use)
smblks: 1
hblks: 2 (chunks from mmap)
hblkhd: 401408 (bytes from mmap)
allmem: 55799808 (bytes from sbrk + mmap)
usmblks: 0
fsmblks: 96
uordblks: 1760864 (bytes used)
fordblks: 53637536 (bytes free)
keepcost: 134944 (bytes in releasable chunk)
rsync[16164] (server receiver) heap statistics:
arena: 58601472 (bytes from sbrk)
ordblks: 434 (chunks not in use)
smblks: 1
hblks: 2 (chunks from mmap)
hblkhd: 401408 (bytes from mmap)
allmem: 59002880 (bytes from sbrk + mmap)
usmblks: 0
fsmblks: 96
uordblks: 1777440 (bytes used)
fordblks: 56824032 (bytes free)
keepcost: 131760 (bytes in releasable chunk)
rsync[16163] (server generator) heap statistics:
arena: 24981504 (bytes from sbrk)
ordblks: 298 (chunks not in use)
smblks: 1
hblks: 2 (chunks from mmap)
hblkhd: 401408 (bytes from mmap)
allmem: 25382912 (bytes from sbrk + mmap)
usmblks: 0
fsmblks: 96
uordblks: 1741776 (bytes used)
fordblks: 23239728 (bytes free)
keepcost: 132688 (bytes in releasable chunk)
Number of files: 92,682 (reg: 83,977, dir: 8,592, link: 113)
Number of created files: 65 (reg: 36, dir: 29)
Number of deleted files: 0
Number of regular files transferred: 36
Total file size: 725,175,978 bytes
Total transferred file size: 1,022,996 bytes
Literal data: 1,022,996 bytes
Matched data: 0 bytes
File list size: 1,310,592
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 4,870,564
Total bytes received: 20,030
sent 4,870,564 bytes received 20,030 bytes 50,159.94 bytes/sec
total size is 725,175,978 speedup is 148.28
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1196) [sender=3.1.2]
Time taken: 97.13713943400217 sec
Source size: 0.726908025 GB
Dest size: 1.7719e-05 GB
Some system logs
dmesg
[17725.959546] usb 1-3: device descriptor read/8, error -110
[17731.335595] usb 1-3: device descriptor read/8, error -110
[17731.443695] usb 1-3: USB disconnect, device number 91
[17731.459678] print_req_error: 15386 callbacks suppressed
[17731.459685] blk_update_request: I/O error, dev sdd, sector 16 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0
[17731.459694] buffer_io_error: 141088 callbacks suppressed
[17731.459697] Buffer I/O error on dev sdd, logical block 2, lost async page write
[17731.459832] blk_update_request: I/O error, dev sdd, sector 32 op 0x1:(WRITE) flags 0x100000 phys_seg 2 prio class 0
[17731.459841] Buffer I/O error on dev sdd, logical block 4, lost async page write
[17731.459850] Buffer I/O error on dev sdd, logical block 5, lost async page write
[17731.459936] blk_update_request: I/O error, dev sdd, sector 16 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[17731.459945] Buffer I/O error on dev sdd, logical block 2, lost async page write
[17731.459980] blk_update_request: I/O error, dev sdd, sector 80 op 0x1:(WRITE) flags 0x100000 phys_seg 12 prio class 0
[17731.459984] Buffer I/O error on dev sdd, logical block 10, lost async page write
[17731.459989] Buffer I/O error on dev sdd, logical block 11, lost async page write
[17731.459993] Buffer I/O error on dev sdd, logical block 12, lost async page write
[17731.459997] Buffer I/O error on dev sdd, logical block 13, lost async page write
[17731.460001] Buffer I/O error on dev sdd, logical block 14, lost async page write
[17731.460006] Buffer I/O error on dev sdd, logical block 15, lost async page write
[17731.460023] blk_update_request: I/O error, dev sdd, sector 32 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[17731.460127] blk_update_request: I/O error, dev sdd, sector 3792424 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0
[17731.460213] blk_update_request: I/O error, dev sdd, sector 3792440 op 0x1:(WRITE) flags 0x100000 phys_seg 2 prio class 0
[17731.460236] blk_update_request: I/O error, dev sdd, sector 3792552 op 0x1:(WRITE) flags 0x0 phys_seg 2 prio class 0
[17731.460258] blk_update_request: I/O error, dev sdd, sector 3794144 op 0x1:(WRITE) flags 0x0 phys_seg 2 prio class 0
[17731.460277] blk_update_request: I/O error, dev sdd, sector 3826880 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
[17731.627498] usb 1-3: new high-speed USB device number 92 using xhci_hcd
[17736.839587] usb 1-3: device descriptor read/64, error -110
[17747.531417] buffer_io_error: 59 callbacks suppressed
[17747.531418] Buffer I/O error on dev sdd, logical block 474055, async page read
[17747.531538] Buffer I/O error on dev sdd, logical block 474055, async page read
[17747.651649] Buffer I/O error on dev sdd, logical block 474055, async page read
[17747.651763] Buffer I/O error on dev sdd, logical block 474055, async page read
[17747.652206] Buffer I/O error on dev sdd, logical block 474055, async page read
[17747.652290] Buffer I/O error on dev sdd, logical block 474055, async page read
[17747.652372] Buffer I/O error on dev sdd, logical block 474055, async page read
[17747.652455] Buffer I/O error on dev sdd, logical block 474055, async page read
[17747.652528] Buffer I/O error on dev sdd, logical block 474055, async page read
[17747.652601] Buffer I/O error on dev sdd, logical block 474055, async page read
[17752.459672] usb 1-3: device descriptor read/64, error -110
[17752.703633] usb 1-3: new high-speed USB device number 93 using xhci_hcd
[17758.091712] usb 1-3: device descriptor read/64, error -110
[17773.707909] usb 1-3: device descriptor read/64, error -110
[17773.815977] usb usb1-port3: attempt power cycle
[17774.227773] usb 1-3: new high-speed USB device number 94 using xhci_hcd
[17779.463982] usb 1-3: device descriptor read/8, error -110
[17784.839933] usb 1-3: device descriptor read/8, error -110
[17785.079874] usb 1-3: new high-speed USB device number 95 using xhci_hcd
[17790.216030] usb 1-3: device descriptor read/8, error -110
[17795.592004] usb 1-3: device descriptor read/8, error -110
[17795.700110] usb usb1-port3: unable to enumerate USB device
/var/log/syslog
Apr 11 15:39:34 SMC-DESKTOP-07 ntfs-3g[14291]: Reading $BITMAP failed: Input/output error
Apr 11 15:39:34 SMC-DESKTOP-07 ntfs-3g[14291]: Failed to allocate clusters: Input/output error
Apr 11 15:39:34 SMC-DESKTOP-07 ntfs-3g[14291]: Failed to allocate (4) clusters for $MFT: Input/output error
Apr 11 15:39:34 SMC-DESKTOP-07 ntfs-3g[14291]: ntfs_attr_pread_i: ntfs_pread failed: Input/output error
Apr 11 15:39:34 SMC-DESKTOP-07 ntfs-3g[14291]: Reading $BITMAP failed: Input/output error
Apr 11 15:39:34 SMC-DESKTOP-07 ntfs-3g[14291]: Failed to allocate clusters: Input/output error
Apr 11 15:39:34 SMC-DESKTOP-07 ntfs-3g[14291]: Failed to allocate (4) clusters for $MFT: Input/output error
Apr 11 15:39:34 SMC-DESKTOP-07 ntfs-3g[14291]: ntfs_attr_pread_i: ntfs_pread failed: Input/output error
Apr 11 15:39:34 SMC-DESKTOP-07 ntfs-3g[14291]: Reading $BITMAP failed: Input/output error
Apr 11 15:39:34 SMC-DESKTOP-07 ntfs-3g[14291]: Failed to allocate clusters: Input/output error
Apr 11 15:39:34 SMC-DESKTOP-07 ntfs-3g[14291]: Failed to allocate (4) clusters for $MFT: Input/output error
Apr 11 15:39:34 SMC-DESKTOP-07 ntfs-3g[14291]: ntfs_attr_pread_i: ntfs_pread failed: Input/output error
Apr 11 15:39:34 SMC-DESKTOP-07 ntfs-3g[14291]: Reading $BITMAP failed: Input/output error
Apr 11 15:39:34 SMC-DESKTOP-07 ntfs-3g[14291]: Failed to allocate clusters: Input/output error
Apr 11 15:39:34 SMC-DESKTOP-07 ntfs-3g[14291]: Failed to allocate (4) clusters for $MFT: Input/output error
Apr 11 15:39:34 SMC-DESKTOP-07 ntfs-3g[14291]: ntfs_attr_pread_i: ntfs_pread failed: Input/output error
Apr 11 15:39:34 SMC-DESKTOP-07 ntfs-3g[14291]: Reading $BITMAP failed: Input/output error
Apr 11 15:39:34 SMC-DESKTOP-07 ntfs-3g[14291]: Failed to allocate clusters: Input/output error
Apr 11 15:39:34 SMC-DESKTOP-07 ntfs-3g[14291]: Failed to allocate (4) clusters for $MFT: Input/output error
Apr 11 15:39:34 SMC-DESKTOP-07 ntfs-3g[14291]: ntfs_attr_pread_i: ntfs_pread failed: Input/output error
Apr 11 15:39:34 SMC-DESKTOP-07 ntfs-3g[14291]: Reading $BITMAP failed: Input/output error
Apr 11 15:39:34 SMC-DESKTOP-07 ntfs-3g[14291]: Failed to allocate clusters: Input/output error
Apr 11 15:39:34 SMC-DESKTOP-07 ntfs-3g[14291]: Failed to allocate (4) clusters for $MFT: Input/output error
Apr 11 15:39:34 SMC-DESKTOP-07 ntfs-3g[14291]: ntfs_attr_pread_i: ntfs_pread failed: Input/output error
Apr 11 15:39:34 SMC-DESKTOP-07 ntfs-3g[14291]: Reading $BITMAP failed: Input/output error
Apr 11 15:39:34 SMC-DESKTOP-07 ntfs-3g[14291]: Failed to allocate clusters: Input/output error
Apr 11 15:39:34 SMC-DESKTOP-07 ntfs-3g[14291]: Failed to allocate (4) clusters for $MFT: Input/output error
Apr 11 15:39:38 SMC-DESKTOP-07 rtkit-daemon[1379]: Supervising 7 threads of 3 processes of 1 users.
Apr 11 15:39:39 SMC-DESKTOP-07 kernel: [17752.459672] usb 1-3: device descriptor read/64, error -110
Apr 11 15:39:39 SMC-DESKTOP-07 kernel: [17752.703633] usb 1-3: new high-speed USB device number 93 using xhci_hcd
Apr 11 15:39:45 SMC-DESKTOP-07 kernel: [17758.091712] usb 1-3: device descriptor read/64, error -110
Apr 11 15:40:00 SMC-DESKTOP-07 kernel: [17773.707909] usb 1-3: device descriptor read/64, error -110
Apr 11 15:40:01 SMC-DESKTOP-07 kernel: [17773.815977] usb usb1-port3: attempt power cycle
Apr 11 15:40:01 SMC-DESKTOP-07 kernel: [17774.227773] usb 1-3: new high-speed USB device number 94 using xhci_hcd
Apr 11 15:40:06 SMC-DESKTOP-07 kernel: [17779.463982] usb 1-3: device descriptor read/8, error -110
Apr 11 15:40:12 SMC-DESKTOP-07 kernel: [17784.839933] usb 1-3: device descriptor read/8, error -110
Apr 11 15:40:12 SMC-DESKTOP-07 kernel: [17785.079874] usb 1-3: new high-speed USB device number 95 using xhci_hcd
Apr 11 15:40:17 SMC-DESKTOP-07 kernel: [17790.216030] usb 1-3: device descriptor read/8, error -110
Apr 11 15:39:46 SMC-DESKTOP-07 rtkit-daemon[1379]: message repeated 3 times: [ Supervising 7 threads of 3 processes of 1 users.]
Apr 11 15:40:19 SMC-DESKTOP-07 systemd-udevd[360]: seq 12441 '/devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0/host7/target7:0:0/7:0:0:0/block/sdd' is taking a long time
Apr 11 15:40:22 SMC-DESKTOP-07 kernel: [17795.592004] usb 1-3: device descriptor read/8, error -110
Apr 11 15:40:22 SMC-DESKTOP-07 kernel: [17795.700110] usb usb1-port3: unable to enumerate USB device
Apr 11 15:40:22 SMC-DESKTOP-07 systemd-udevd[17326]: inotify_add_watch(9, /dev/sdd, 10) failed: No such file or directory
Apr 11 15:40:22 SMC-DESKTOP-07 udisksd[921]: Cleaning up mount point /media/smc-desktop-07/16gb (device 8:48 no longer exists)
Apr 11 15:40:22 SMC-DESKTOP-07 upowerd[1341]: unhandled action 'unbind' on /sys/devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0
Apr 11 15:40:22 SMC-DESKTOP-07 upowerd[1341]: unhandled action 'unbind' on /sys/devices/pci0000:00/0000:00:14.0/usb1/1-3
Apr 11 15:40:22 SMC-DESKTOP-07 systemd[1]: Stopping Clean the /media/smc-desktop-07/16gb mount point...
Apr 11 15:40:22 SMC-DESKTOP-07 systemd[1]: Stopped Clean the /media/smc-desktop-07/16gb mount point.