What are the benchmark details and full results used for our NVMe servers?
What tools we used for benchmarking the NVMe
In order to benchmark our powerful NVMe devices, we have used two Linux tools - dd and fio. You can see below the test commands and the full output of the run commands.
dd tests
Write - speed
Test command: dd if=/dev/zero of=benchmark bs=64K count=32K conv=fdatasync
Command output (result):
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 1.50309 s, 1.4 GB/s
Test command: dd if=/dev/zero of=dd.test bs=64K count=256K conv=fdatasync
Command output (result):
262144+0 records in
262144+0 records out
17179869184 bytes (17 GB, 16 GiB) copied, 8.46257 s, 2.0 GB/s
Read - speed
Test command: dd if=dd.test of=/dev/null bs=64k count=32k
Command output (result):
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 0.229655 s, 9.4 GB/s
Test command: dd if=dd.test of=/dev/null bs=64k count=128k
Command output (result):
131072+0 records in
131072+0 records out
8589934592 bytes (8.6 GB, 8.0 GiB) copied, 1.58605 s, 5.4 GB/s
fio tests
Write - speed
Test command: fio --name=write_throughput --numjobs=8 --size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 --verify=0 --bs=1M --iodepth=64 --rw=write --group_reporting=1
Command output (result):
Starting 8 processes
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
Jobs: 8 (f=8): [W(8)][100.0%][r=0KiB/s,w=3164MiB/s][r=0,w=3164 IOPS][eta 00m:00s]
write_throughput: (groupid=0, jobs=8): err= 0: pid=10912: Mon Jan 11 04:07:51 2021
write: IOPS=3259, BW=3268MiB/s (3427MB/s)(192GiB/60057msec)
slat (usec): min=31, max=319727, avg=2448.12, stdev=8914.52
clat (msec): min=7, max=655, avg=154.37, stdev=59.93
lat (msec): min=15, max=677, avg=156.82, stdev=60.43
clat percentiles (msec):
| 1.00th=[ 54], 5.00th=[ 77], 10.00th=[ 89], 20.00th=[ 108],
| 30.00th=[ 123], 40.00th=[ 134], 50.00th=[ 146], 60.00th=[ 159],
| 70.00th=[ 174], 80.00th=[ 194], 90.00th=[ 226], 95.00th=[ 262],
| 99.00th=[ 355], 99.50th=[ 401], 99.90th=[ 502], 99.95th=[ 558],
| 99.99th=[ 625]
bw ( KiB/s): min=102400, max=675714, per=12.53%, avg=419474.84, stdev=86076.92, samples=960
iops : min= 100, max= 659, avg=409.42, stdev=84.05, samples=960
lat (msec) : 10=0.01%, 20=0.01%, 50=0.72%, 100=14.67%, 250=78.65%
lat (msec) : 500=6.10%, 750=0.10%
cpu : usr=1.61%, sys=2.31%, ctx=39138, majf=0, minf=20
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=103.1%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwt: total=0,195771,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=3268MiB/s (3427MB/s), 3268MiB/s-3268MiB/s (3427MB/s-3427MB/s), io=192GiB (206GB), run=60057-60057msec
Disk stats (read/write):
vda: ios=0/202497, merge=0/1395, ticks=0/7814080, in_queue=7020948, util=98.61%
Write - IOPS
Test command: fio --name=write_iops --size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=64 --rw=randwrite --group_reporting=1
Command output (result):
write_iops: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=330MiB/s][r=0,w=84.4k IOPS][eta 00m:00s]
write_iops: (groupid=0, jobs=1): err= 0: pid=10649: Mon Jan 11 04:01:01 2021
write: IOPS=56.6k, BW=221MiB/s (232MB/s)(12.0GiB/60001msec)
slat (nsec): min=1869, max=20186k, avg=12428.09, stdev=156839.62
clat (usec): min=49, max=36464, avg=1115.69, stdev=1311.93
lat (usec): min=63, max=36469, avg=1128.48, stdev=1322.40
clat percentiles (usec):
| 1.00th=[ 318], 5.00th=[ 562], 10.00th=[ 644], 20.00th=[ 725],
| 30.00th=[ 791], 40.00th=[ 848], 50.00th=[ 906], 60.00th=[ 963],
| 70.00th=[ 1029], 80.00th=[ 1123], 90.00th=[ 1319], 95.00th=[ 1713],
| 99.00th=[10028], 99.50th=[10814], 99.90th=[13304], 99.95th=[14746],
| 99.99th=[17957]
bw ( KiB/s): min=149592, max=376248, per=99.57%, avg=225608.39, stdev=50841.34, samples=119
iops : min=37398, max=94062, avg=56402.08, stdev=12710.35, samples=119
lat (usec) : 50=0.01%, 100=0.01%, 250=0.22%, 500=3.05%, 750=20.89%
lat (usec) : 1000=41.81%
lat (msec) : 2=30.35%, 4=1.77%, 10=0.93%, 20=0.99%, 50=0.01%
cpu : usr=7.34%, sys=48.26%, ctx=644059, majf=0, minf=3
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=103.7%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwt: total=0,3398809,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=221MiB/s (232MB/s), 221MiB/s-221MiB/s (232MB/s-232MB/s), io=12.0GiB (13.9GB), run=60001-60001msec
Disk stats (read/write):
vda: ios=0/3997028, merge=0/1826387, ticks=0/1777860, in_queue=1527644, util=94.14%
Read - speed
Test command: fio --name=read_throughput --numjobs=8 --size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 --verify=0 --bs=1M --iodepth=64 --rw=read --group_reporting=1
Command output (result):
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
Jobs: 8 (f=8): [R(8)][100.0%][r=6739MiB/s,w=0KiB/s][r=6738,w=0 IOPS][eta 00m:00s]
read_throughput: (groupid=0, jobs=8): err= 0: pid=10586: Mon Jan 11 03:44:00 2021
read: IOPS=8440, BW=8449MiB/s (8859MB/s)(495GiB/60039msec)
slat (usec): min=18, max=288674, avg=940.87, stdev=5243.52
clat (msec): min=2, max=438, avg=59.70, stdev=37.25
lat (msec): min=3, max=438, avg=60.64, stdev=37.57
clat percentiles (msec):
| 1.00th=[ 14], 5.00th=[ 20], 10.00th=[ 24], 20.00th=[ 31],
| 30.00th=[ 38], 40.00th=[ 45], 50.00th=[ 52], 60.00th=[ 59],
| 70.00th=[ 69], 80.00th=[ 82], 90.00th=[ 105], 95.00th=[ 131],
| 99.00th=[ 201], 99.50th=[ 230], 99.90th=[ 292], 99.95th=[ 317],
| 99.99th=[ 355]
bw ( MiB/s): min= 303, max= 1776, per=12.50%, avg=1056.48, stdev=249.55, samples=960
iops : min= 303, max= 1776, avg=1056.32, stdev=249.56, samples=960
lat (msec) : 4=0.01%, 10=0.15%, 20=5.17%, 50=42.73%, 100=40.71%
lat (msec) : 250=11.03%, 500=0.31%
cpu : usr=0.25%, sys=6.34%, ctx=66229, majf=0, minf=20
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=103.1%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwt: total=506756,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=8449MiB/s (8859MB/s), 8449MiB/s-8449MiB/s (8859MB/s-8859MB/s), io=495GiB (532GB), run=60039-60039msec
Disk stats (read/write):
vda: ios=523996/293, merge=0/66, ticks=7648240/3904, in_queue=6653724, util=99.01%
Read - IOPS
Test command: fio --name=read_iops --size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=64 --rw=randread --group_reporting=1
Command output (result):
Starting 1 process
Jobs: 1 (f=0): [f(1)][100.0%][r=586MiB/s,w=0KiB/s][r=150k,w=0 IOPS][eta 00m:00s]
read_iops: (groupid=0, jobs=1): err= 0: pid=10535: Mon Jan 11 03:37:21 2021
read: IOPS=149k, BW=580MiB/s (609MB/s)(34.0GiB/60001msec)
slat (nsec): min=1680, max=236800, avg=3171.88, stdev=3210.98
clat (usec): min=72, max=3868, avg=426.08, stdev=122.45
lat (usec): min=80, max=3870, avg=429.56, stdev=122.60
clat percentiles (usec):
| 1.00th=[ 217], 5.00th=[ 255], 10.00th=[ 281], 20.00th=[ 326],
| 30.00th=[ 363], 40.00th=[ 396], 50.00th=[ 424], 60.00th=[ 453],
| 70.00th=[ 482], 80.00th=[ 515], 90.00th=[ 562], 95.00th=[ 603],
| 99.00th=[ 701], 99.50th=[ 791], 99.90th=[ 1483], 99.95th=[ 1680],
| 99.99th=[ 1876]
bw ( KiB/s): min=447400, max=640128, per=99.97%, avg=594167.11, stdev=23663.97, samples=119
iops : min=111850, max=160032, avg=148541.72, stdev=5915.99, samples=119
lat (usec) : 100=0.01%, 250=4.23%, 500=71.79%, 750=23.40%, 1000=0.18%
lat (msec) : 2=0.39%, 4=0.01%
cpu : usr=19.61%, sys=61.60%, ctx=150554, majf=0, minf=2
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=103.2%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwt: total=8915060,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=580MiB/s (609MB/s), 580MiB/s-580MiB/s (609MB/s-609MB/s), io=34.0GiB (36.5GB), run=60001-60001msec
Disk stats (read/write):
vda: ios=9161062/58, merge=0/7, ticks=2747560/12, in_queue=2714548, util=99.87%