I 'm trying to clone a zfs file system pool/u01 to a new file system called newpool/u01 using following commands
zfs list
zfs snapshot pool/u01@new
zfs send pool/u01@new | zfs -F receive newpool/u01
Its a 100G file system snapshot and copied to same server on different pool and took like 5hrs to complete and the problem is that the overall throught put i get is very very poor. I 'm not able to figure out why .
Generally it should be 50-60M/sec but not in my case , can anyone suggest what could be the reason and how can i improve this one.
Thanks in Advance.
tell a bit more... what kind of server? how are the disks connected and how are the pools build? from the commands send and receive we can not say anything... would all be speculations.
Install mbuffer and run:
zfs send pool/u01@new | mbuffer -s 128k -m 2G -o -| zfs -F receive newpool/u01
Source: Using mbuffer to speed up slow zfs send | zfs receive
Hi Jillagre,
I cannot install this on my production server without n no. of approvals.
Thanks.
---------- Post updated at 09:34 AM ---------- Previous update was at 09:10 AM ----------
Its a M4000 server with pools configured following
bash-3.00$ zpool status prd_newpool
pool: prd_newpool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
prd_newpool ONLINE 0 0 0
c5t6006016027C02C006E8E04308D17E211d0 ONLINE 0 0 0
errors: No known data errors
-bash-3.00$ zpool status prd_pool
pool: prd_pool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
prd_pool ONLINE 0 0 0
c5t60060160EEB12C007A5677518D3CE111d0 ONLINE 0 0 0
c5t60060160EEB12C0092C024408D3CE111d0 ONLINE 0 0 0
c5t60060160EEB12C0094C024408D3CE111d0 ONLINE 0 0 0
c5t60060160EEB12C0096C024408D3CE111d0 ONLINE 0 0 0
errors: No known data errors
iostat o/p shows following
pool alloc free read write read write
-------------------------------------- ----- ----- ----- ----- ----- -----
prd_newpool 117G 81.6G 0 0 0 0
c5t6006016027C02C006E8E04308D17E211d0 117G 81.6G 0 0 0 0
-------------------------------------- ----- ----- ----- ----- ----- -----
prd_pool 1.19T 715G 1.04K 0 9.18M 25.6K
c5t60060160EEB12C007A5677518D3CE111d0 296G 152G 214 0 1.90M 8.80K
c5t60060160EEB12C0092C024408D3CE111d0 309G 187G 279 0 2.45M 5.60K
c5t60060160EEB12C0094C024408D3CE111d0 308G 188G 279 0 2.42M 5.60K
c5t60060160EEB12C0096C024408D3CE111d0 308G 188G 296 0 2.40M 5.60K
-------------------------------------- ----- ----- ----- ----- ----- -----
capacity operations bandwidth
pool alloc free read write read write
-------------------------------------- ----- ----- ----- ----- ----- -----
prd_newpool 117G 81.6G 0 112 0 13.9M
c5t6006016027C02C006E8E04308D17E211d0 117G 81.6G 0 112 0 13.9M
-------------------------------------- ----- ----- ----- ----- ----- -----
prd_pool 1.19T 715G 682 1 7.42M 51.2K
c5t60060160EEB12C007A5677518D3CE111d0 296G 152G 155 0 1.92M 14.4K
c5t60060160EEB12C0092C024408D3CE111d0 309G 187G 182 0 1.91M 9.60K
c5t60060160EEB12C0094C024408D3CE111d0 308G 188G 190 0 1.74M 12.8K
c5t60060160EEB12C0096C024408D3CE111d0 308G 188G 153 0 1.86M 14.4K
-------------------------------------- ----- ----- ----- ----- ----- -----
capacity operations bandwidth
pool alloc free read write read write
-------------------------------------- ----- ----- ----- ----- ----- -----
prd_newpool 118G 81.4G 0 216 0 25.8M
c5t6006016027C02C006E8E04308D17E211d0 118G 81.4G 0 216 0 25.8M
-------------------------------------- ----- ----- ----- ----- ----- -----
prd_pool 1.19T 715G 966 1 17.1M 50.4K
c5t60060160EEB12C007A5677518D3CE111d0 296G 152G 232 0 4.18M 7.20K
c5t60060160EEB12C0092C024408D3CE111d0 309G 187G 249 0 4.34M 11.2K
c5t60060160EEB12C0094C024408D3CE111d0 308G 188G 262 0 4.31M 12.8K
c5t60060160EEB12C0096C024408D3CE111d0 308G 188G 222 0 4.32M 19.2K
-------------------------------------- ----- ----- ----- ----- ----- -----
capacity operations bandwidth
pool alloc free read write read write
-------------------------------------- ----- ----- ----- ----- ----- -----
prd_newpool 118G 81.4G 0 0 0 0
c5t6006016027C02C006E8E04308D17E211d0 118G 81.4G 0 0 0 0
-------------------------------------- ----- ----- ----- ----- ----- -----
prd_pool 1.19T 715G 1.25K 0 18.7M 25.6K
c5t60060160EEB12C007A5677518D3CE111d0 296G 152G 317 0 4.71M 5.60K
c5t60060160EEB12C0092C024408D3CE111d0 309G 187G 306 0 4.70M 5.60K
c5t60060160EEB12C0094C024408D3CE111d0 308G 188G 325 0 4.64M 5.60K
c5t60060160EEB12C0096C024408D3CE111d0 308G 188G 327 0 4.69M 8.80K
-------------------------------------- ----- ----- ----- ----- ----- -----
capacity operations bandwidth
pool alloc free read write read write
-------------------------------------- ----- ----- ----- ----- ----- -----
prd_newpool 118G 81.4G 0 0 0 0
c5t6006016027C02C006E8E04308D17E211d0 118G 81.4G 0 0 0 0
-------------------------------------- ----- ----- ----- ----- ----- -----
prd_pool 1.19T 715G 729 1 10.5M 50.4K
c5t60060160EEB12C007A5677518D3CE111d0 296G 152G 193 0 2.65M 17.6K
c5t60060160EEB12C0092C024408D3CE111d0 309G 187G 174 0 2.63M 12.8K
c5t60060160EEB12C0094C024408D3CE111d0 308G 188G 166 0 2.63M 14.4K
c5t60060160EEB12C0096C024408D3CE111d0 308G 188G 193 0 2.62M 5.60K
-------------------------------------- ----- ----- ----- ----- ----- -----
achenle
October 30, 2012, 10:09am
5
Could you post the results of:
iostat -sndzx 2
Run that while you're doing the ZFS send/receive.
Then the wisest way would be for you to ask for these approvals. There is no much point trying to find other causes and/or seek different solutions while that one has proved to be very successful.
c5t6006016027C02C006E8E04308D17E211d0 is the disk from the pool which is the target pool where zfs receive is running.
144.4 1.0 1522.2 13.5 0.0 0.3 0.0 2.0 0 9 c5t60060160EEB12C007A5677518D3CE111d0
0.5 1117.6 0.5 121058.6 0.0 4.8 0.0 4.3 1 60 c5t6006016027C02C006E8E04308D17E211d0
530.8 1.0 6686.0 10.4 0.0 1.1 0.0 2.0 0 23 c5t60060160EEB12C0092C024408D3CE111d0
520.9 1.0 6947.7 27.0 0.0 1.1 0.0 2.1 0 21 c5t60060160EEB12C0094C024408D3CE111d0
415.1 0.0 6938.1 0.0 0.0 2.1 0.0 5.1 0 55 c5t60060160EEB12C0096C024408D3CE111d0
465.9 0.0 6837.4 0.0 0.0 1.1 0.0 2.4 0 23 c5t60060160EEB12C007A5677518D3CE111d0
0.0 895.0 0.0 109071.0 0.0 3.7 0.0 4.2 1 45 c5t6006016027C02C006E8E04308D17E211d0
594.1 0.0 8065.5 0.0 0.0 1.6 0.0 2.7 0 35 c5t60060160EEB12C0092C024408D3CE111d0
658.0 0.0 8284.4 0.0 0.0 1.1 0.0 1.6 0 23 c5t60060160EEB12C0094C024408D3CE111d0
495.6 1.0 7908.7 10.0 0.0 1.8 0.0 3.6 0 42 c5t60060160EEB12C0096C024408D3CE111d0
554.6 1.0 7773.0 18.0 0.0 1.4 0.0 2.4 0 32 c5t60060160EEB12C007A5677518D3CE111d0
642.6 1.0 8369.3 14.0 0.0 1.3 0.0 2.1 0 31 c5t60060160EEB12C0092C024408D3CE111d0
669.6 1.0 8486.9 22.0 0.0 1.2 0.0 1.8 0 24 c5t60060160EEB12C0094C024408D3CE111d0
536.8 1.0 8346.1 14.0 0.0 2.5 0.0 4.7 0 46 c5t60060160EEB12C0096C024408D3CE111d0
593.2 1.0 8226.7 14.0 0.0 1.3 0.0 2.2 0 28 c5t60060160EEB12C007A5677518D3CE111d0
645.9 1.0 8482.2 26.1 0.0 1.3 0.0 2.0 0 29 c5t60060160EEB12C0092C024408D3CE111d0
667.9 1.0 8293.6 10.0 0.0 1.2 0.0 1.8 0 26 c5t60060160EEB12C0094C024408D3CE111d0
558.2 0.0 8523.8 0.0 0.0 2.1 0.0 3.8 0 41 c5t60060160EEB12C0096C024408D3CE111d0
571.7 0.0 8304.1 0.0 0.0 1.7 0.0 3.0 0 34 c5t60060160EEB12C007A5677518D3CE111d0
557.7 0.0 8027.1 0.0 0.0 1.3 0.0 2.4 0 31 c5t60060160EEB12C0092C024408D3CE111d0
549.7 0.0 7813.6 0.0 0.0 1.7 0.0 3.1 0 34 c5t60060160EEB12C0094C024408D3CE111d0
549.7 1.0 8166.2 10.0 0.0 1.8 0.0 3.2 0 36 c5t60060160EEB12C0096C024408D3CE111d0
571.2 1.0 8105.0 18.0 0.0 1.1 0.0 1.9 0 24 c5t60060160EEB12C007A5677518D3CE111d0
821.3 1.0 10725.2 22.0 0.0 1.6 0.0 1.9 0 32 c5t60060160EEB12C0092C024408D3CE111d0
725.7 1.0 10210.3 14.0 0.0 1.4 0.0 1.9 0 27 c5t60060160EEB12C0094C024408D3CE111d0
653.7 1.0 10444.3 14.0 0.0 1.5 0.0 2.2 0 34 c5t60060160EEB12C0096C024408D3CE111d0
696.7 1.0 10167.7 14.0 0.0 2.0 0.0 2.9 0 38 c5t60060160EEB12C007A5677518D3CE111d0
391.2 1.0 4949.9 14.0 0.0 1.1 0.0 2.9 0 29 c5t60060160EEB12C0092C024408D3CE111d0
406.1 1.0 4793.5 22.0 0.0 0.9 0.0 2.2 0 22 c5t60060160EEB12C0094C024408D3CE111d0
321.7 0.0 4777.3 0.0 0.0 2.5 0.0 7.8 0 54 c5t60060160EEB12C0096C024408D3CE111d0
403.6 0.0 4873.7 0.0 0.0 1.0 0.0 2.4 0 23 c5t60060160EEB12C007A5677518D3CE111d0
498.3 0.0 7018.1 0.0 0.0 1.0 0.0 2.0 0 22 c5t60060160EEB12C0092C024408D3CE111d0
497.8 0.0 6537.8 0.0 0.0 0.9 0.0 1.9 0 19 c5t60060160EEB12C0094C024408D3CE111d0
469.8 1.0 6941.5 10.0 0.0 1.6 0.0 3.3 0 32 c5t60060160EEB12C0096C024408D3CE111d0
500.3 1.0 6716.9 18.0 0.0 2.1 0.0 4.1 0 47 c5t60060160EEB12C007A5677518D3CE111d0
600.7 1.0 8832.4 10.0 0.0 1.5 0.0 2.4 0 29 c5t60060160EEB12C0092C024408D3CE111d0
620.2 1.0 8989.2 26.0 0.0 1.5 0.0 2.4 0 36 c5t60060160EEB12C0094C024408D3CE111d0
592.7 1.0 8906.8 18.0 0.0 1.8 0.0 3.0 0 44 c5t60060160EEB12C0096C024408D3CE111d0
575.2 1.0 8172.5 10.0 0.0 1.1 0.0 1.9 0 25 c5t60060160EEB12C007A5677518D3CE111d0
266.9 1.0 2901.8 14.0 0.0 1.1 0.0 4.2 0 24 c5t60060160EEB12C0092C024408D3CE111d0
267.4 1.0 3002.9 22.0 0.0 0.4 0.0 1.4 0 10 c5t60060160EEB12C0094C024408D3CE111d0
168.2 0.0 2661.0 0.0 0.0 2.0 0.0 12.2 0 56 c5t60060160EEB12C0096C024408D3CE111d0
185.8 0.0 2563.6 0.0 0.0 0.9 0.0 4.6 0 18 c5t60060160EEB12C007A5677518D3CE111d0
361.5 0.0 3781.1 0.0 0.0 0.9 0.0 2.5 0 34 c5t60060160EEB12C0092C024408D3CE111d0
379.6 0.0 3817.9 0.0 0.0 0.6 0.0 1.5 0 19 c5t60060160EEB12C0094C024408D3CE111d0
392.6 1.0 4048.2 18.0 0.0 0.9 0.0 2.2 0 25 c5t60060160EEB12C0096C024408D3CE111d0
356.5 1.0 3620.1 10.0 0.0 1.0 0.0 2.8 0 40 c5t60060160EEB12C007A5677518D3CE111d0
322.2 1.0 2057.7 22.0 0.0 0.5 0.0 1.4 0 18 c5t60060160EEB12C0092C024408D3CE111d0
375.7 1.0 2065.4 14.0 0.0 0.4 0.0 1.0 0 16 c5t60060160EEB12C0094C024408D3CE111d0
315.7 1.0 1699.5 22.0 0.0 0.5 0.0 1.5 0 25 c5t60060160EEB12C0096C024408D3CE111d0
232.8 1.0 1651.6 6.0 0.0 1.2 0.0 5.2 0 52 c5t60060160EEB12C007A5677518D3CE111d0
505.9 1.0 4825.4 22.0 0.0 0.9 0.0 1.8 0 27 c5t60060160EEB12C0092C024408D3CE111d0
495.9 1.0 4953.1 14.0 0.0 0.8 0.0 1.6 0 24 c5t60060160EEB12C0094C024408D3CE111d0
377.4 0.0 4523.2 0.0 0.0 1.2 0.0 3.2 0 35 c5t60060160EEB12C0096C024408D3CE111d0
450.4 0.0 4440.7 0.0 0.0 1.1 0.0 2.4 0 35 c5t60060160EEB12C007A5677518D3CE111d0
418.0 25.5 3333.3 107.1 0.0 0.8 0.0 1.9 0 26 c5t60060160EEB12C0092C024408D3CE111d0
519.9 21.5 3585.5 69.4 0.0 0.8 0.0 1.4 0 27 c5t60060160EEB12C0094C024408D3CE111d0
474.0 16.5 3445.9 56.7 0.0 0.9 0.0 1.9 0 37 c5t60060160EEB12C0096C024408D3CE111d0
361.1 10.0 2607.1 67.9 0.0 0.7 0.0 1.8 0 32 c5t60060160EEB12C007A5677518D3CE111d0
330.3 1.0 2915.0 22.0 0.0 1.0 0.0 3.1 0 30 c5t60060160EEB12C0092C024408D3CE111d0
368.3 1.0 2978.1 14.0 0.0 0.6 0.0 1.7 0 20 c5t60060160EEB12C0094C024408D3CE111d0
324.8 1.0 3148.7 18.0 0.0 1.3 0.0 4.1 0 43 c5t60060160EEB12C0096C024408D3CE111d0
286.2 1.0 2801.4 10.0 0.0 1.0 0.0 3.4 0 30 c5t60060160EEB12C007A5677518D3CE111d0
0.0 706.0 0.0 73351.1 0.0 5.5 0.0 7.7 0 58 c5t6006016027C02C006E8E04308D17E211d0
318.0 1.0 1682.3 17.8 0.0 0.5 0.0 1.6 0 25 c5t60060160EEB12C0092C024408D3CE111d0
314.0 1.0 1666.7 17.8 0.0 0.5 0.0 1.5 0 23 c5t60060160EEB12C0094C024408D3CE111d0
220.2 0.0 1144.3 0.0 0.0 0.9 0.0 4.0 0 30 c5t60060160EEB12C0096C024408D3CE111d0
193.6 0.0 1243.5 0.0 0.0 1.2 0.0 6.3 0 33 c5t60060160EEB12C007A5677518D3CE111d0
0.0 1069.3 0.0 136244.6 0.0 9.3 0.0 8.7 1 100 c5t6006016027C02C006E8E04308D17E211d0
182.0 0.0 1113.7 0.0 0.0 0.6 0.0 3.2 0 23 c5t60060160EEB12C0092C024408D3CE111d0
224.6 0.0 1352.7 0.0 0.0 0.4 0.0 1.9 0 17 c5t60060160EEB12C0094C024408D3CE111d0
197.2 1.0 1155.5 18.2 0.0 0.5 0.0 2.5 0 32 c5t60060160EEB12C0096C024408D3CE111d0
123.2 1.0 911.2 10.1 0.0 1.1 0.0 8.5 0 41 c5t60060160EEB12C007A5677518D3CE111d0
0.0 1014.5 0.0 129080.0 0.0 9.4 0.0 9.3 1 100 c5t6006016027C02C006E8E04308D17E211d0
211.2 3.0 1109.0 65.9 0.0 0.6 0.0 2.6 0 26 c5t60060160EEB12C0092C024408D3CE111d0
232.1 3.0 1161.7 41.9 0.0 0.3 0.0 1.4 0 17 c5t60060160EEB12C0094C024408D3CE111d0
160.3 3.0 1089.3 53.9 0.0 1.5 0.0 9.1 0 50 c5t60060160EEB12C0096C024408D3CE111d0
181.7 4.0 1146.0 47.9 0.0 0.5 0.0 2.5 0 21 c5t60060160EEB12C007A5677518D3CE111d0
0.0 61.9 0.0 5270.5 0.0 0.4 0.0 7.2 0 5 c5t6006016027C02C006E8E04308D17E211d0
312.0 0.0 1690.6 0.0 0.0 0.7 0.0 2.1 0 36 c5t60060160EEB12C0092C024408D3CE111d0
308.5 0.0 1501.3 0.0 0.0 0.3 0.0 1.1 0 15 c5t60060160EEB12C0094C024408D3CE111d0
234.9 0.0 1477.7 0.0 0.0 1.0 0.0 4.2 0 35 c5t60060160EEB12C0096C024408D3CE111d0
235.9 0.0 1550.1 0.0 0.0 0.4 0.0 1.8 0 24 c5t60060160EEB12C007A5677518D3CE111d0
284.0 1.0 2115.4 18.0 0.0 0.7 0.0 2.5 0 25 c5t60060160EEB12C0092C024408D3CE111d0
302.5 1.0 2359.9 18.0 0.0 0.8 0.0 2.6 0 31 c5t60060160EEB12C0094C024408D3CE111d0
305.5 1.0 2196.2 22.0 0.0 0.9 0.0 3.0 0 34 c5t60060160EEB12C0096C024408D3CE111d0
239.5 1.0 2132.9 6.0 0.0 0.6 0.0 2.6 0 27 c5t60060160EEB12C007A5677518D3CE111d0
293.0 1.0 3435.7 18.0 0.0 1.4 0.0 4.8 0 33 c5t60060160EEB12C0092C024408D3CE111d0
270.0 1.0 3370.0 18.0 0.0 1.3 0.0 4.8 0 37 c5t60060160EEB12C0094C024408D3CE111d0
241.5 1.0 3190.4 22.0 0.0 1.0 0.0 4.0 0 30 c5t60060160EEB12C0096C024408D3CE111d0
218.5 1.0 3237.7 6.0 0.0 1.0 0.0 4.5 0 34 c5t60060160EEB12C007A5677518D3CE111d0
283.0 0.0 2620.7 0.0 0.0 1.2 0.0 4.2 0 34 c5t60060160EEB12C0092C024408D3CE111d0
253.0 0.0 2715.4 0.0 0.0 1.2 0.0 4.6 0 33 c5t60060160EEB12C0094C024408D3CE111d0
261.5 0.0 2677.2 0.0 0.0 1.0 0.0 4.0 0 30 c5t60060160EEB12C0096C024408D3CE111d0
224.5 0.0 2228.4 0.0 0.0 0.7 0.0 2.9 0 18 c5t60060160EEB12C007A5677518D3CE111d0
224.5 1.0 1420.7 22.0 0.0 0.3 0.0 1.5 0 17 c5t60060160EEB12C0092C024408D3CE111d0
220.5 1.0 1473.9 14.0 0.0 0.6 0.0 2.9 0 33 c5t60060160EEB12C0094C024408D3CE111d0
202.0 1.0 1176.7 26.0 0.0 0.3 0.0 1.7 0 14 c5t60060160EEB12C0096C024408D3CE111d0
168.0 0.0 1215.2 0.0 0.0 1.1 0.0 6.4 0 46 c5t60060160EEB12C007A5677518D3CE111d0
202.0 1.0 1159.2 18.0 0.0 0.6 0.0 2.9 0 20 c5t60060160EEB12C0092C024408D3CE111d0
181.0 1.0 1137.7 10.0 0.0 0.5 0.0 3.0 0 23 c5t60060160EEB12C0094C024408D3CE111d0
208.0 1.0 1469.0 18.0 0.0 1.1 0.0 5.1 0 53 c5t60060160EEB12C0096C024408D3CE111d0
154.0 1.0 1058.0 18.0 0.0 0.3 0.0 2.1 0 13 c5t60060160EEB12C007A5677518D3CE111d0
303.3 0.0 1502.9 0.0 0.0 0.5 0.0 1.5 0 20 c5t60060160EEB12C0092C024408D3CE111d0
230.0 0.0 1438.8 0.0 0.0 0.6 0.0 2.5 0 24 c5t60060160EEB12C0094C024408D3CE111d0
192.0 0.0 1189.2 0.0 0.0 0.9 0.0 4.9 0 43 c5t60060160EEB12C0096C024408D3CE111d0
281.8 0.0 1374.0 0.0 0.0 0.5 0.0 1.7 0 25 c5t60060160EEB12C007A5677518D3CE111d0
107.7 1.0 566.9 22.0 0.0 0.1 0.0 1.3 0 7 c5t60060160EEB12C0092C024408D3CE111d0
77.1 1.0 565.4 22.0 0.0 1.1 0.0 13.8 0 36 c5t60060160EEB12C0094C024408D3CE111d0
95.6 1.0 578.6 6.0 0.0 0.9 0.0 9.8 0 38 c5t60060160EEB12C0096C024408D3CE111d0
86.1 1.0 514.3 14.0 0.0 1.2 0.0 14.0 0 44 c5t60060160EEB12C007A5677518D3CE111d0
285.3 1.0 1540.7 26.0 0.0 0.5 0.0 1.8 0 20 c5t60060160EEB12C0092C024408D3CE111d0
282.3 1.0 1662.3 14.0 0.0 0.6 0.0 2.0 0 24 c5t60060160EEB12C0094C024408D3CE111d0
276.8 1.0 1679.3 14.0 0.0 1.0 0.0 3.7 0 31 c5t60060160EEB12C0096C024408D3CE111d0
283.8 1.0 1574.7 10.0 0.0 0.7 0.0 2.4 0 38 c5t60060160EEB12C007A5677518D3CE111d0
175.4 0.0 996.7 0.0 0.0 0.6 0.0 3.5 0 26 c5t60060160EEB12C0092C024408D3CE111d0
213.3 0.0 829.8 0.0 0.0 0.4 0.0 1.9 0 19 c5t60060160EEB12C0094C024408D3CE111d0
175.4 0.0 630.7 0.0 0.0 0.4 0.0 2.0 0 20 c5t60060160EEB12C0096C024408D3CE111d0
226.8 0.0 1054.9 0.0 0.0 0.9 0.0 3.9 0 46 c5t60060160EEB12C007A5677518D3CE111d0
116.6 51.5 540.7 448.9 0.0 0.2 0.0 1.4 0 12 c5t60060160EEB12C0092C024408D3CE111d0
110.6 35.5 586.5 333.0 0.0 0.5 0.0 3.6 0 40 c5t60060160EEB12C0094C024408D3CE111d0
98.6 20.0 623.0 261.7 0.0 0.4 0.0 3.3 0 28 c5t60060160EEB12C0096C024408D3CE111d0
98.6 29.5 555.7 362.8 0.0 1.6 0.0 12.2 0 43 c5t60060160EEB12C007A5677518D3CE111d0
206.0 1.0 1092.6 14.0 0.0 0.4 0.0 2.1 0 25 c5t60060160EEB12C0092C024408D3CE111d0
199.5 1.0 1156.7 18.0 0.0 0.4 0.0 1.8 0 18 c5t60060160EEB12C0094C024408D3CE111d0
196.5 1.0 1128.7 10.0 0.0 0.4 0.0 1.8 0 16 c5t60060160EEB12C0096C024408D3CE111d0
189.0 1.0 1165.4 21.9 0.0 0.9 0.0 4.8 0 38 c5t60060160EEB12C007A5677518D3CE111d0
0.0 667.3 0.0 42128.5 0.0 3.8 0.0 5.7 0 41 c5t6006016027C02C006E8E04308D17E211d0
424.6 0.0 4171.4 0.0 0.0 0.7 0.0 1.6 0 21 c5t60060160EEB12C0092C024408D3CE111d0
464.2 0.0 4388.4 0.0 0.0 1.0 0.0 2.2 0 31 c5t60060160EEB12C0094C024408D3CE111d0
437.1 0.0 4053.2 0.0 0.0 1.0 0.0 2.4 0 35 c5t60060160EEB12C0096C024408D3CE111d0
419.1 0.0 3844.2 0.0 0.0 1.0 0.0 2.5 0 31 c5t60060160EEB12C007A5677518D3CE111d0
0.0 488.7 0.0 46941.7 0.0 3.4 0.0 7.0 0 38 c5t6006016027C02C006E8E04308D17E211d0
437.8 1.0 3608.7 14.0 0.0 0.7 0.0 1.7 0 27 c5t60060160EEB12C0092C024408D3CE111d0
397.3 1.0 3399.6 26.0 0.0 1.0 0.0 2.5 0 36 c5t60060160EEB12C0094C024408D3CE111d0
312.3 0.0 3185.9 0.0 0.0 1.0 0.0 3.1 0 37 c5t60060160EEB12C0096C024408D3CE111d0
360.8 1.0 3619.2 22.0 0.0 0.8 0.0 2.3 0 28 c5t60060160EEB12C007A5677518D3CE111d0
189.7 1.0 1262.8 18.0 0.0 0.5 0.0 2.6 0 19 c5t60060160EEB12C0092C024408D3CE111d0
192.2 1.0 1123.2 10.0 0.0 0.4 0.0 2.0 0 23 c5t60060160EEB12C0094C024408D3CE111d0
174.2 1.0 1017.4 18.0 0.0 0.5 0.0 2.9 0 27 c5t60060160EEB12C0096C024408D3CE111d0
176.2 1.0 1091.0 18.0 0.0 1.2 0.0 6.7 0 47 c5t60060160EEB12C007A5677518D3CE111d0
332.5 0.0 2983.5 0.0 0.0 0.8 0.0 2.5 0 31 c5t60060160EEB12C0092C024408D3CE111d0
270.4 0.0 2700.8 0.0 0.0 0.9 0.0 3.4 0 28 c5t60060160EEB12C0094C024408D3CE111d0
288.4 0.0 3095.4 0.0 0.0 0.6 0.0 2.2 0 19 c5t60060160EEB12C0096C024408D3CE111d0
277.9 0.0 2734.4 0.0 0.0 0.9 0.0 3.2 0 40 c5t60060160EEB12C007A5677518D3CE111d0
185.1 1.0 1587.4 18.0 0.0 0.3 0.0 1.7 0 12 c5t60060160EEB12C0092C024408D3CE111d0
205.6 1.0 1691.2 10.0 0.0 0.3 0.0 1.6 0 15 c5t60060160EEB12C0094C024408D3CE111d0
87.5 1.0 1081.4 26.0 0.0 0.6 0.0 6.3 0 20 c5t60060160EEB12C0096C024408D3CE111d0
130.1 1.0 1606.7 10.0 0.0 2.4 0.0 18.0 0 67 c5t60060160EEB12C007A5677518D3CE111d0
206.1 1.0 1304.7 14.0 0.0 0.4 0.0 1.9 0 23 c5t60060160EEB12C0092C024408D3CE111d0
203.6 1.0 1146.2 14.0 0.0 0.4 0.0 2.2 0 27 c5t60060160EEB12C0094C024408D3CE111d0
246.1 1.0 1455.1 18.0 0.0 0.6 0.0 2.4 0 29 c5t60060160EEB12C0096C024408D3CE111d0
146.6 1.0 853.3 18.0 0.0 0.9 0.0 6.4 0 32 c5t60060160EEB12C007A5677518D3CE111d0
370.1 0.0 2136.9 0.0 0.0 0.5 0.0 1.4 0 25 c5t60060160EEB12C0092C024408D3CE111d0
348.6 0.0 2144.7 0.0 0.0 0.5 0.0 1.5 0 25 c5t60060160EEB12C0094C024408D3CE111d0
383.1 0.0 1952.1 0.0 0.0 0.6 0.0 1.5 0 24 c5t60060160EEB12C0096C024408D3CE111d0
306.1 0.0 1745.8 0.0 0.0 0.7 0.0 2.4 0 39 c5t60060160EEB12C007A5677518D3CE111d0
150.0 1.0 1126.6 18.0 0.0 0.3 0.0 1.8 0 12 c5t60060160EEB12C0092C024408D3CE111d0
158.0 1.0 1244.1 10.0 0.0 0.4 0.0 2.7 0 22 c5t60060160EEB12C0094C024408D3CE111d0
150.5 1.0 1110.8 14.0 0.0 0.3 0.0 2.3 0 15 c5t60060160EEB12C0096C024408D3CE111d0
130.0 1.0 858.2 22.0 0.0 2.4 0.0 18.0 0 62 c5t60060160EEB12C007A5677518D3CE111d0
322.7 1.0 2393.6 14.0 0.0 0.7 0.0 2.2 0 25 c5t60060160EEB12C0092C024408D3CE111d0
308.2 1.0 2323.7 14.0 0.0 0.6 0.0 2.0 0 27 c5t60060160EEB12C0094C024408D3CE111d0
273.7 1.0 2298.7 14.0 0.0 0.5 0.0 2.0 0 17 c5t60060160EEB12C0096C024408D3CE111d0
282.7 1.0 2165.6 22.0 0.0 1.1 0.0 4.0 0 50 c5t60060160EEB12C007A5677518D3CE111d0
262.2 0.0 1750.0 0.0 0.0 0.7 0.0 2.7 0 35 c5t60060160EEB12C0092C024408D3CE111d0
298.2 0.0 1774.7 0.0 0.0 0.6 0.0 2.0 0 25 c5t60060160EEB12C0094C024408D3CE111d0
316.8 0.0 1875.8 0.0 0.0 1.1 0.0 3.6 0 34 c5t60060160EEB12C0096C024408D3CE111d0
203.7 0.0 1587.1 0.0 0.0 0.5 0.0 2.3 0 24 c5t60060160EEB12C007A5677518D3CE111d0
541.6 1.0 5460.4 18.0 0.0 0.9 0.0 1.6 0 24 c5t60060160EEB12C0092C024408D3CE111d0
439.2 1.0 5466.2 10.0 0.0 1.0 0.0 2.3 0 32 c5t60060160EEB12C0094C024408D3CE111d0
426.7 1.0 5678.0 26.0 0.0 1.3 0.0 3.1 0 34 c5t60060160EEB12C0096C024408D3CE111d0
363.2 1.0 5056.0 10.0 0.0 1.7 0.0 4.6 0 37 c5t60060160EEB12C007A5677518D3CE111d0
491.2 1.0 5019.1 14.0 0.0 0.8 0.0 1.6 0 20 c5t60060160EEB12C0092C024408D3CE111d0
411.7 1.0 5098.4 14.0 0.0 1.7 0.0 4.0 0 38 c5t60060160EEB12C0094C024408D3CE111d0
353.2 1.0 5158.0 18.0 0.0 1.4 0.0 3.9 0 34 c5t60060160EEB12C0096C024408D3CE111d0
309.6 1.0 4694.2 18.0 0.0 1.8 0.0 5.7 0 41 c5t60060160EEB12C007A5677518D3CE111d0
264.9 0.0 3130.0 0.0 0.0 1.1 0.0 4.1 0 30 c5t60060160EEB12C0092C024408D3CE111d0
224.4 0.0 3085.3 0.0 0.0 1.4 0.0 6.4 0 33 c5t60060160EEB12C0094C024408D3CE111d0
236.9 0.0 3048.3 0.0 0.0 1.3 0.0 5.7 0 31 c5t60060160EEB12C0096C024408D3CE111d0
213.0 0.0 2885.8 0.0 0.0 1.1 0.0 5.1 0 29 c5t60060160EEB12C007A5677518D3CE111d0
395.5 5.5 5185.4 23.2 0.0 1.0 0.0 2.5 0 19 c5t60060160EEB12C0092C024408D3CE111d0
350.0 22.0 5147.7 152.7 0.0 1.6 0.0 4.3 0 35 c5t60060160EEB12C0094C024408D3CE111d0
334.0 24.5 5132.9 105.7 0.0 1.5 0.0 4.2 0 37 c5t60060160EEB12C0096C024408D3CE111d0
342.5 17.0 4659.2 58.2 0.0 1.8 0.0 5.0 0 33 c5t60060160EEB12C007A5677518D3CE111d0
293.7 1.0 3585.9 9.9 0.0 0.7 0.0 2.4 0 18 c5t60060160EEB12C0092C024408D3CE111d0
242.2 1.0 3453.6 17.8 0.0 1.2 0.0 5.1 0 24 c5t60060160EEB12C0094C024408D3CE111d0
222.4 1.0 3453.9 17.8 0.0 1.3 0.0 5.9 0 37 c5t60060160EEB12C0096C024408D3CE111d0
189.7 1.0 3271.3 17.8 0.0 1.7 0.0 9.0 0 35 c5t60060160EEB12C007A5677518D3CE111d0
1.0 1092.6 1.2 62896.5 0.0 5.4 0.0 4.9 1 58 c5t6006016027C02C006E8E04308D17E211d0
522.4 0.0 6991.1 0.0 0.0 1.3 0.0 2.4 0 30 c5t60060160EEB12C0092C024408D3CE111d0
530.4 0.0 6832.0 0.0 0.0 1.2 0.0 2.2 0 26 c5t60060160EEB12C0094C024408D3CE111d0
536.0 0.0 7089.9 0.0 0.0 1.3 0.0 2.5 0 30 c5t60060160EEB12C0096C024408D3CE111d0
475.6 0.0 6651.8 0.0 0.0 1.3 0.0 2.7 0 32 c5t60060160EEB12C007A5677518D3CE111d0
0.5 850.3 0.5 102811.1 0.0 2.8 0.0 3.3 0 37 c5t6006016027C02C006E8E04308D17E211d0
747.5 1.0 8251.4 18.0 0.0 1.4 0.0 1.8 0 28 c5t60060160EEB12C0092C024408D3CE111d0
611.8 1.0 7956.0 10.0 0.0 1.4 0.0 2.4 0 31 c5t60060160EEB12C0094C024408D3CE111d0
649.9 1.0 8371.6 22.0 0.0 1.5 0.0 2.3 0 32 c5t60060160EEB12C0096C024408D3CE111d0
the iostat output seconds the need for mbuffer... btw zpool iostat would be a better tool for monitoring io on a zpool:http://docs.oracle.com/cd/E19253-01/819-5461/gammt/index.html
OP says mbuffer isn't going to be allowed.
Running
zfs send ... > /some/file/name
where the file is on a known FAST file system would help isolate if the issue is in the send or the receive. Run the send, see how fast it is, then run receive off that file and see how fast that is.
Maybe some useful data points can be generated.
And there might even be some way to use a file and some standard utilities to duplicate the functionality of mbuffer. Maybe something as simple as starting the send to a file, waiting a few minutes, then starting the receive might work if the send always runs faster than the receive and the receive won't hit EOF before the send is done.
Achelen that seems to be good way to pin down what is slowing ..
followng is the zpool iostat o/p
bash-3.00$ zpool iostat -v prd_pool prd_newpool 5
capacity operations bandwidth
pool alloc free read write read write
-------------------------------------- ----- ----- ----- ----- ----- -----
prd_newpool 98.1G 101G 0 2 307 219K
c5t6006016027C02C006E8E04308D17E211d0 98.1G 101G 0 2 307 219K
-------------------------------------- ----- ----- ----- ----- ----- -----
prd_pool 1.19T 714G 185 18 8.53M 1.09M
c5t60060160EEB12C007A5677518D3CE111d0 296G 152G 44 4 2.07M 266K
c5t60060160EEB12C0092C024408D3CE111d0 309G 187G 46 4 2.16M 282K
c5t60060160EEB12C0094C024408D3CE111d0 308G 188G 47 4 2.15M 282K
c5t60060160EEB12C0096C024408D3CE111d0 308G 188G 47 4 2.15M 282K
-------------------------------------- ----- ----- ----- ----- ----- -----
capacity operations bandwidth
pool alloc free read write read write
-------------------------------------- ----- ----- ----- ----- ----- -----
prd_newpool 98.1G 101G 0 0 0 0
c5t6006016027C02C006E8E04308D17E211d0 98.1G 101G 0 0 0 0
-------------------------------------- ----- ----- ----- ----- ----- -----
prd_pool 1.19T 714G 1.30K 1 29.7M 36.8K
c5t60060160EEB12C007A5677518D3CE111d0 296G 152G 379 0 7.45M 11.2K
c5t60060160EEB12C0092C024408D3CE111d0 309G 187G 248 0 7.36M 11.2K
c5t60060160EEB12C0094C024408D3CE111d0 308G 188G 348 0 7.43M 5.60K
c5t60060160EEB12C0096C024408D3CE111d0 308G 188G 353 0 7.46M 8.79K
-------------------------------------- ----- ----- ----- ----- ----- -----
capacity operations bandwidth
pool alloc free read write read write
-------------------------------------- ----- ----- ----- ----- ----- -----
prd_newpool 98.1G 101G 0 0 0 0
c5t6006016027C02C006E8E04308D17E211d0 98.1G 101G 0 0 0 0
-------------------------------------- ----- ----- ----- ----- ----- -----
prd_pool 1.19T 714G 830 0 19.7M 39.2K
c5t60060160EEB12C007A5677518D3CE111d0 296G 152G 237 0 4.88M 10.4K
c5t60060160EEB12C0092C024408D3CE111d0 309G 187G 147 0 4.92M 7.20K
c5t60060160EEB12C0094C024408D3CE111d0 308G 188G 198 0 4.86M 16.0K
c5t60060160EEB12C0096C024408D3CE111d0 308G 188G 246 0 5.04M 5.60K
-------------------------------------- ----- ----- ----- ----- ----- -----
capacity operations bandwidth
pool alloc free read write read write
-------------------------------------- ----- ----- ----- ----- ----- -----
prd_newpool 98.1G 101G 0 280 0 26.0M
c5t6006016027C02C006E8E04308D17E211d0 98.1G 101G 0 280 0 26.0M
-------------------------------------- ----- ----- ----- ----- ----- -----
prd_pool 1.19T 714G 477 23 13.4M 162K
c5t60060160EEB12C007A5677518D3CE111d0 296G 152G 144 4 3.46M 53.9K
c5t60060160EEB12C0092C024408D3CE111d0 309G 187G 86 8 3.31M 53.3K
c5t60060160EEB12C0094C024408D3CE111d0 308G 188G 106 6 3.21M 28.3K
c5t60060160EEB12C0096C024408D3CE111d0 308G 188G 139 4 3.45M 26.9K
-------------------------------------- ----- ----- ----- ----- ----- -----
capacity operations bandwidth
pool alloc free read write read write
-------------------------------------- ----- ----- ----- ----- ----- -----
prd_newpool 98.4G 101G 0 203 0 24.0M
c5t6006016027C02C006E8E04308D17E211d0 98.4G 101G 0 203 0 24.0M
-------------------------------------- ----- ----- ----- ----- ----- -----
prd_pool 1.19T 714G 984 0 26.0M 25.6K
c5t60060160EEB12C007A5677518D3CE111d0 296G 152G 233 0 6.21M 5.60K
c5t60060160EEB12C0092C024408D3CE111d0 309G 187G 185 0 6.45M 8.79K
c5t60060160EEB12C0094C024408D3CE111d0 308G 188G 269 0 6.75M 5.60K
c5t60060160EEB12C0096C024408D3CE111d0 308G 188G 296 0 6.54M 5.60K
-------------------------------------- ----- ----- ----- ----- ----- -----