I got some advices from a person who wrote his thesis on the subject of benchmarking:
- benchmark with the applications that are like the later used applications.
- test with concurrent i/o-requests(if that's your scenario, and it is like that almost always).
- test with small block sizes, as this will be the realistic work load for the storage in my case.
- test with virtual machines and with network, so it will be like the i/o when the system is used in production.
I'm already testing with small block sizes. Concurrent jobs testing is running at the moment. Real-world testing will be done at some later point.
--- Post updated at 04:53 PM ---
- Performance Base Line of the system
This benchmark is to demonstrate the actual speed of the used system. It's in no way relevant for the later workload and just to make sure there storage system is generally performing without major trouble.
Interesting: RAID-5 has slower write speeds than expected. zfs-RAIDZ which is similar in it's data distributin is considerably faster.
Single-Disk, Sequential Read, Single Threaded, Test with 1M Block-Size. Bandwidth in KB/s
singledisk.zfs.seq_read.bs_1M.compressed.run_1.json 129088
singledisk.zfs.seq_read.bs_1M.run_1.json 129365
singledisk.ext3.seq_read.bs_1M.run_1.json 140170
singledisk.ext3.seq_read.bs_1M.lvm.run_1.json 140197
singledisk.ext4.seq_read.bs_1M.lvm.run_1.json 148060
singledisk.ext4.seq_read.bs_1M.run_1.json 148154
singledisk.btrfs.seq_read.bs_1M.lvm.run_1.json 151971
singledisk.btrfs.seq_read.bs_1M.run_1.json 154073
Single-Disk, Sequential Write, Single Threaded, Test with 1M Block-Size. Bandwidth in KB/s
singledisk.btrfs.seq_write.bs_1M.run_1.json 112133
singledisk.btrfs.seq_write.bs_1M.lvm.run_1.json 115764
singledisk.zfs.seq_write.bs_1M.compressed.run_1.json 130645
singledisk.zfs.seq_write.bs_1M.run_1.json 132107
singledisk.ext3.seq_write.bs_1M.lvm.run_1.json 132526
singledisk.ext3.seq_write.bs_1M.run_1.json 132902
singledisk.ext4.seq_write.bs_1M.lvm.run_1.json 146049
singledisk.ext4.seq_write.bs_1M.run_1.json 146220
Single-Disk, Sequential Read, Single Threaded, Test with 4K Block-Size. IOPS
singledisk.zfs.seq_read.bs_4k.run_1.json 33963
singledisk.zfs.seq_read.bs_4k.compressed.run_1.json 34625
singledisk.ext3.seq_read.bs_4k.run_1.json 36023
singledisk.ext3.seq_read.bs_4k.lvm.run_1.json 36028
singledisk.ext4.seq_read.bs_4k.lvm.run_1.json 37882
singledisk.ext4.seq_read.bs_4k.run_1.json 38013
singledisk.btrfs.seq_read.bs_4k.lvm.run_1.json 38643
singledisk.btrfs.seq_read.bs_4k.run_1.json 38647
Single-Disk, Sequential Write, Single Threaded, Test with 4K Block-Size. IOPS
singledisk.btrfs.seq_write.bs_4k.run_1.json 890
singledisk.btrfs.seq_write.bs_4k.lvm.run_1.json 895
singledisk.zfs.seq_write.bs_4k.run_1.json 3026
singledisk.zfs.seq_write.bs_4k.compressed.run_1.json 3189
singledisk.ext3.seq_write.bs_4k.run_1.json 3465
singledisk.ext3.seq_write.bs_4k.lvm.run_1.json 3476
singledisk.ext4.seq_write.bs_4k.lvm.run_1.json 6972
singledisk.ext4.seq_write.bs_4k.run_1.json 7237
Multi-Disk, Sequential Read, Single Threaded, Test with 1M Block-Size. Bandwidth in KB/s
raid10.zfs.seq_read.bs_1M.compressed.run_1.json 267917
raid10.zfs.seq_read.bs_1M.raid10.run_1.json 287920
raid5.ext3.seq_read.bs_1M.raid5.run_1.json 291303
raid5.ext4.seq_read.bs_1M.raid5.run_1.json 294709
raid10btr_native.btrfs.seq_read.bs_1M.raid10btr_run_1.json 298280
raid10.btrfs.seq_read.bs_1M.raid10.run_1.json 310551
raid10.btrfs.seq_read.bs_1M.lvm.run_1.json 317500
raid10.ext3.seq_read.bs_1M.raid10.run_1.json 319551
raid5.btrfs.seq_read.bs_1M.raid5.run_1.json 319678
raid10.ext3.seq_read.bs_1M.lvm.run_1.json 327557
raidz.zfs.seq_read.bs_1M.run_1.json 329564
raid5.ext4.seq_read.bs_1M.lvm.run_1.json 330541
raid5.ext3.seq_read.bs_1M.lvm.run_1.json 335769
raidz.zfs.seq_read.bs_1M.compressed.run_1.json 350659
raid10.ext4.seq_read.bs_1M.lvm.run_1.json 354560
raid10.ext4.seq_read.bs_1M.raid10.run_1.json 355618
raid5.btrfs.seq_read.bs_1M.lvm.run_1.json 383868
Multi-Disk, Sequential Write, Single Threaded, Test with 1M Block-Size. Bandwidth in KB/s
raid5.btrfs.seq_write.bs_1M.lvm.run_1.json 63088
raid5.btrfs.seq_write.bs_1M.raid5.run_1.json 63716
raid5.ext4.seq_write.bs_1M.lvm.run_1.json 68732
raid5.ext4.seq_write.bs_1M.raid5.run_1.json 70874
raid5.ext3.seq_write.bs_1M.lvm.run_1.json 73033
raid5.ext3.seq_write.bs_1M.raid5.run_1.json 73329
raid10.zfs.seq_write.bs_1M.compressed.run_1.json 197587
raid10.zfs.seq_write.bs_1M.raid10.run_1.json 202224
raid10.btrfs.seq_write.bs_1M.lvm.run_1.json 211278
raid10.btrfs.seq_write.bs_1M.raid10.run_1.json 217614
raid10.ext3.seq_write.bs_1M.lvm.run_1.json 221382
raid10.ext3.seq_write.bs_1M.raid10.run_1.json 225781
raid10btr_native.btrfs.seq_write.bs_1M.raid10btr_run_1.json 244794
raid10.ext4.seq_write.bs_1M.raid10.run_1.json 254144
raid10.ext4.seq_write.bs_1M.lvm.run_1.json 254237
raidz.zfs.seq_write.bs_1M.compressed.run_1.json 258104
raidz.zfs.seq_write.bs_1M.run_1.json 276822
Multi-Disk, Sequential Read, Single Threaded, Test with 4K Block-Size. IOPS
raid10.ext4.seq_read.bs_4k.raid10.run_1.json 11616
raid10.btrfs.seq_read.bs_4k.raid10.run_1.json 28261
raid10.ext4.seq_read.bs_4k.lvm.run_1.json 32650
raid10.ext3.seq_read.bs_4k.raid10.run_1.json 34177
raid10.zfs.seq_read.bs_4k.compressed.run_1.json 58822
raid10.zfs.seq_read.bs_4k.raid10.run_1.json 64036
raid5.ext3.seq_read.bs_4k.raid5.run_1.json 70986
raid5.ext4.seq_read.bs_4k.raid5.run_1.json 72429
raid10btr_native.btrfs.seq_read.bs_4k.raid10btr_run_1.json 74755
raid5.btrfs.seq_read.bs_4k.raid5.run_1.json 76245
raid10.btrfs.seq_read.bs_4k.lvm.run_1.json 77020
raid10.ext3.seq_read.bs_4k.lvm.run_1.json 82203
raid5.ext3.seq_read.bs_4k.lvm.run_1.json 83717
raidz.zfs.seq_read.bs_4k.run_1.json 85503
raidz.zfs.seq_read.bs_4k.compressed.run_1.json 87909
raid5.ext4.seq_read.bs_4k.lvm.run_1.json 88718
raid5.btrfs.seq_read.bs_4k.lvm.run_1.json 96451
Multi-Disk, Sequential Write, Single Threaded, Test with 4K Block-Size. IOPS
raid5.btrfs.seq_write.bs_4k.lvm.run_1.json 371
raid5.btrfs.seq_write.bs_4k.raid5.run_1.json 374
raid10.btrfs.seq_write.bs_4k.raid10.run_1.json 1238
raid10.btrfs.seq_write.bs_4k.lvm.run_1.json 1255
raid5.ext3.seq_write.bs_4k.lvm.run_1.json 1714
raid5.ext3.seq_write.bs_4k.raid5.run_1.json 1762
raid10btr_native.btrfs.seq_write.bs_4k.raid10btr_run_1.json 1846
raid5.ext4.seq_write.bs_4k.lvm.run_1.json 2313
raid5.ext4.seq_write.bs_4k.raid5.run_1.json 2408
raid10.zfs.seq_write.bs_4k.raid10.run_1.json 3256
raidz.zfs.seq_write.bs_4k.compressed.run_1.json 3381
raidz.zfs.seq_write.bs_4k.run_1.json 3469
raid10.zfs.seq_write.bs_4k.compressed.run_1.json 3638
raid10.ext3.seq_write.bs_4k.lvm.run_1.json 4165
raid10.ext3.seq_write.bs_4k.raid10.run_1.json 4373
raid10.ext4.seq_write.bs_4k.lvm.run_1.json 5020
raid10.ext4.seq_write.bs_4k.raid10.run_1.json 5442
--- Post updated at 04:56 PM ---
Why do you think that? I do not understand that RAID3 would perform better. As I understand it, it should be nearly the same. Only the parity goes to one dedicated disk and that disk is used heavily.
EDIT: Ahh. I possibly understand. RAID3 uses Byte-Level-Striping instead of Block-Level_Striping of RAID4/5 which may be calculated faster? Unfortunately Linux Software RAID does not support RAID3.
--- Post updated at 05:25 PM ---
-
First Insight: LVM seems not to impact read/write throughput or iops performance
Check the numbers by comparing the neighbor rows with and without lvm with the same other specs. The numbers of those pairs do only differ very little. I'll still test and watch the lvm performance readings, but I'll not report them any more, except there is some worth mentioning.
Single-Disk, Random Read, Single Threaded, Test with 4K Block-Size. IOPS
singledisk.btrfs.random_read.bs_4k.run_3.json 144
singledisk.btrfs.random_read.bs_4k.lvm.run_3.json 146
singledisk.ext3.random_read.bs_4k.run_3.json 150
singledisk.ext3.random_read.bs_4k.lvm.run_3.json 149
singledisk.ext4.random_read.bs_4k.run_3.json 147
singledisk.ext4.random_read.bs_4k.lvm.run_3.json 150
Single-Disk, Random Write, Single Threaded, Test with 4K Block-Size. IOPS
singledisk.btrfs.random_write.bs_4k.run_3.json 192
singledisk.btrfs.random_write.bs_4k.lvm.run_3.json 190
singledisk.ext3.random_write.bs_4k.run_3.json 290
singledisk.ext3.random_write.bs_4k.lvm.run_3.json 290
singledisk.ext4.random_write.bs_4k.lvm.run_3.json 304
singledisk.ext4.random_write.bs_4k.run_3.json 305
Multi-Disk, Random Read, Single Threaded, Test with 4K Block-Size. IOPS
raid10.ext4.random_read.bs_4k.lvm.run_1.json 66
raid10.ext4.random_read.bs_4k.raid10.run_1.json 66
raid10.ext3.random_read.bs_4k.raid10.run_1.json 149
raid10.ext3.random_read.bs_4k.lvm.run_1.json 150
raid10.btrfs.random_read.bs_4k.raid10.run_1.json 153
raid10.btrfs.random_read.bs_4k.lvm.run_1.json 151
raid5.ext4.random_read.bs_4k.raid5.run_1.json 154
raid5.ext4.random_read.bs_4k.lvm.run_1.json 153
raid5.ext3.random_read.bs_4k.raid5.run_1.json 155
raid5.ext3.random_read.bs_4k.lvm.run_1.json 156
raid5.btrfs.random_read.bs_4k.raid5.run_1.json 158
raid5.btrfs.random_read.bs_4k.lvm.run_1.json 159
Multi-Disk, Random Write, Single Threaded, Test with 4K Block-Size. IOPS
raid5.btrfs.random_write.bs_4k.raid5.run_1.json 64
raid5.btrfs.random_write.bs_4k.lvm.run_1.json 65
raid5.ext4.random_write.bs_4k.lvm.run_1.json 67
raid5.ext4.random_write.bs_4k.raid5.run_1.json 68
raid5.ext3.random_write.bs_4k.raid5.run_1.json 78
raid5.ext3.random_write.bs_4k.lvm.run_1.json 78
raid10.btrfs.random_write.bs_4k.lvm.run_1.json 311
raid10.btrfs.random_write.bs_4k.raid10.run_1.json 313
raid10.ext4.random_write.bs_4k.raid10.run_1.json 465
raid10.ext4.random_write.bs_4k.lvm.run_1.json 471
raid10.ext3.random_write.bs_4k.raid10.run_1.json 570
raid10.ext3.random_write.bs_4k.lvm.run_1.json 559
Multi-Disk, Random Read, Single Threaded, Test with 4K Block-Size. Bandwidth in KB/s
raid10.btrfs.random_read.bs_4k.lvm.run_1.json 604
raid10.btrfs.random_read.bs_4k.raid10.run_1.json 613
raid10.ext3.random_read.bs_4k.lvm.run_1.json 600
raid10.ext3.random_read.bs_4k.raid10.run_1.json 594
raid10.ext4.random_read.bs_4k.lvm.run_1.json 265
raid10.ext4.random_read.bs_4k.raid10.run_1.json 265
raid5.btrfs.random_read.bs_4k.lvm.run_1.json 636
raid5.btrfs.random_read.bs_4k.raid5.run_1.json 630
raid5.ext3.random_read.bs_4k.lvm.run_1.json 622
raid5.ext3.random_read.bs_4k.raid5.run_1.json 620
raid5.ext4.random_read.bs_4k.lvm.run_1.json 610
raid5.ext4.random_read.bs_4k.raid5.run_1.json 617
Multi-Disk, Random Write, Single Threaded, Test with 4K Block-Size. Bandwidth in KB/s
raid10.btrfs.random_write.bs_4k.lvm.run_1.json 1245
raid10.btrfs.random_write.bs_4k.raid10.run_1.json 1250
raid10.ext3.random_write.bs_4k.lvm.run_1.json 2234
raid10.ext3.random_write.bs_4k.raid10.run_1.json 2279
raid10.ext4.random_write.bs_4k.lvm.run_1.json 1884
raid10.ext4.random_write.bs_4k.raid10.run_1.json 1858
raid5.btrfs.random_write.bs_4k.lvm.run_1.json 258
raid5.btrfs.random_write.bs_4k.raid5.run_1.json 257
raid5.ext3.random_write.bs_4k.lvm.run_1.json 312
raid5.ext3.random_write.bs_4k.raid5.run_1.json 312
raid5.ext4.random_write.bs_4k.lvm.run_1.json 268
raid5.ext4.random_write.bs_4k.raid5.run_1.json 270