Table of contents
Found a mistake? Report it to us by selecting the wrong word/s and press CTRL + ALT + M. Thank you!
The article did not help you?
Contact us

What are the benchmark details and full results used for our NVMe servers?

What tools we used for benchmarking the NVMe

In order to benchmark our powerful NVMe devices, we have used two Linux tools - dd and fio. You can see below the test commands and the full output of the run commands.

 

dd tests

Write - speed

Test command: dd if=/dev/zero of=benchmark bs=64K count=32K conv=fdatasync 

Command output (result):

32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 1.50309 s, 1.4 GB/s

 

Test command: dd if=/dev/zero of=dd.test bs=64K count=256K conv=fdatasync

Command output (result):

262144+0 records in
262144+0 records out
17179869184 bytes (17 GB, 16 GiB) copied, 8.46257 s, 2.0 GB/s

 

Read  - speed

Test command: dd if=dd.test of=/dev/null bs=64k count=32k

Command output (result):

32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 0.229655 s, 9.4 GB/s

 

Test command: dd if=dd.test of=/dev/null bs=64k count=128k

Command output (result):

131072+0 records in
131072+0 records out
8589934592 bytes (8.6 GB, 8.0 GiB) copied, 1.58605 s, 5.4 GB/s

 

fio tests

Write - speed

Test command: fio --name=write_throughput --numjobs=8 --size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 --verify=0 --bs=1M --iodepth=64 --rw=write --group_reporting=1

Command output (result):

Starting 8 processes
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
write_throughput: Laying out IO file (1 file / 10240MiB)
Jobs: 8 (f=8): [W(8)][100.0%][r=0KiB/s,w=3164MiB/s][r=0,w=3164 IOPS][eta 00m:00s]
write_throughput: (groupid=0, jobs=8): err= 0: pid=10912: Mon Jan 11 04:07:51 2021
  write: IOPS=3259, BW=3268MiB/s (3427MB/s)(192GiB/60057msec)
    slat (usec): min=31, max=319727, avg=2448.12, stdev=8914.52
    clat (msec): min=7, max=655, avg=154.37, stdev=59.93
     lat (msec): min=15, max=677, avg=156.82, stdev=60.43
    clat percentiles (msec):
     |  1.00th=[   54],  5.00th=[   77], 10.00th=[   89], 20.00th=[  108],
     | 30.00th=[  123], 40.00th=[  134], 50.00th=[  146], 60.00th=[  159],
     | 70.00th=[  174], 80.00th=[  194], 90.00th=[  226], 95.00th=[  262],
     | 99.00th=[  355], 99.50th=[  401], 99.90th=[  502], 99.95th=[  558],
     | 99.99th=[  625]
   bw (  KiB/s): min=102400, max=675714, per=12.53%, avg=419474.84, stdev=86076.92, samples=960
   iops        : min=  100, max=  659, avg=409.42, stdev=84.05, samples=960
  lat (msec)   : 10=0.01%, 20=0.01%, 50=0.72%, 100=14.67%, 250=78.65%
  lat (msec)   : 500=6.10%, 750=0.10%
  cpu          : usr=1.61%, sys=2.31%, ctx=39138, majf=0, minf=20
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=103.1%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=0,195771,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=3268MiB/s (3427MB/s), 3268MiB/s-3268MiB/s (3427MB/s-3427MB/s), io=192GiB (206GB), run=60057-60057msec

Disk stats (read/write):
  vda: ios=0/202497, merge=0/1395, ticks=0/7814080, in_queue=7020948, util=98.61%

 

Write - IOPS

Test command: fio --name=write_iops --size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=64 --rw=randwrite --group_reporting=1

Command output (result):

write_iops: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=330MiB/s][r=0,w=84.4k IOPS][eta 00m:00s]
write_iops: (groupid=0, jobs=1): err= 0: pid=10649: Mon Jan 11 04:01:01 2021
  write: IOPS=56.6k, BW=221MiB/s (232MB/s)(12.0GiB/60001msec)
    slat (nsec): min=1869, max=20186k, avg=12428.09, stdev=156839.62
    clat (usec): min=49, max=36464, avg=1115.69, stdev=1311.93
     lat (usec): min=63, max=36469, avg=1128.48, stdev=1322.40
    clat percentiles (usec):
     |  1.00th=[  318],  5.00th=[  562], 10.00th=[  644], 20.00th=[  725],
     | 30.00th=[  791], 40.00th=[  848], 50.00th=[  906], 60.00th=[  963],
     | 70.00th=[ 1029], 80.00th=[ 1123], 90.00th=[ 1319], 95.00th=[ 1713],
     | 99.00th=[10028], 99.50th=[10814], 99.90th=[13304], 99.95th=[14746],
     | 99.99th=[17957]
   bw (  KiB/s): min=149592, max=376248, per=99.57%, avg=225608.39, stdev=50841.34, samples=119
   iops        : min=37398, max=94062, avg=56402.08, stdev=12710.35, samples=119
  lat (usec)   : 50=0.01%, 100=0.01%, 250=0.22%, 500=3.05%, 750=20.89%
  lat (usec)   : 1000=41.81%
  lat (msec)   : 2=30.35%, 4=1.77%, 10=0.93%, 20=0.99%, 50=0.01%
  cpu          : usr=7.34%, sys=48.26%, ctx=644059, majf=0, minf=3
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=103.7%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=0,3398809,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=221MiB/s (232MB/s), 221MiB/s-221MiB/s (232MB/s-232MB/s), io=12.0GiB (13.9GB), run=60001-60001msec

Disk stats (read/write):
  vda: ios=0/3997028, merge=0/1826387, ticks=0/1777860, in_queue=1527644, util=94.14%

 

Read - speed

Test command: fio --name=read_throughput --numjobs=8 --size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 --verify=0 --bs=1M --iodepth=64 --rw=read --group_reporting=1

Command output (result):

read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
read_throughput: Laying out IO file (1 file / 10240MiB)
Jobs: 8 (f=8): [R(8)][100.0%][r=6739MiB/s,w=0KiB/s][r=6738,w=0 IOPS][eta 00m:00s]
read_throughput: (groupid=0, jobs=8): err= 0: pid=10586: Mon Jan 11 03:44:00 2021
   read: IOPS=8440, BW=8449MiB/s (8859MB/s)(495GiB/60039msec)
    slat (usec): min=18, max=288674, avg=940.87, stdev=5243.52
    clat (msec): min=2, max=438, avg=59.70, stdev=37.25
     lat (msec): min=3, max=438, avg=60.64, stdev=37.57
    clat percentiles (msec):
     |  1.00th=[   14],  5.00th=[   20], 10.00th=[   24], 20.00th=[   31],
     | 30.00th=[   38], 40.00th=[   45], 50.00th=[   52], 60.00th=[   59],
     | 70.00th=[   69], 80.00th=[   82], 90.00th=[  105], 95.00th=[  131],
     | 99.00th=[  201], 99.50th=[  230], 99.90th=[  292], 99.95th=[  317],
     | 99.99th=[  355]
   bw (  MiB/s): min=  303, max= 1776, per=12.50%, avg=1056.48, stdev=249.55, samples=960
   iops        : min=  303, max= 1776, avg=1056.32, stdev=249.56, samples=960
  lat (msec)   : 4=0.01%, 10=0.15%, 20=5.17%, 50=42.73%, 100=40.71%
  lat (msec)   : 250=11.03%, 500=0.31%
  cpu          : usr=0.25%, sys=6.34%, ctx=66229, majf=0, minf=20
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=103.1%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=506756,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=8449MiB/s (8859MB/s), 8449MiB/s-8449MiB/s (8859MB/s-8859MB/s), io=495GiB (532GB), run=60039-60039msec

Disk stats (read/write):
  vda: ios=523996/293, merge=0/66, ticks=7648240/3904, in_queue=6653724, util=99.01%

 

Read - IOPS

Test command: fio --name=read_iops --size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 --verify=0 --bs=4K --iodepth=64 --rw=randread --group_reporting=1

Command output (result):

Starting 1 process
Jobs: 1 (f=0): [f(1)][100.0%][r=586MiB/s,w=0KiB/s][r=150k,w=0 IOPS][eta 00m:00s]
read_iops: (groupid=0, jobs=1): err= 0: pid=10535: Mon Jan 11 03:37:21 2021
   read: IOPS=149k, BW=580MiB/s (609MB/s)(34.0GiB/60001msec)
    slat (nsec): min=1680, max=236800, avg=3171.88, stdev=3210.98
    clat (usec): min=72, max=3868, avg=426.08, stdev=122.45
     lat (usec): min=80, max=3870, avg=429.56, stdev=122.60
    clat percentiles (usec):
     |  1.00th=[  217],  5.00th=[  255], 10.00th=[  281], 20.00th=[  326],
     | 30.00th=[  363], 40.00th=[  396], 50.00th=[  424], 60.00th=[  453],
     | 70.00th=[  482], 80.00th=[  515], 90.00th=[  562], 95.00th=[  603],
     | 99.00th=[  701], 99.50th=[  791], 99.90th=[ 1483], 99.95th=[ 1680],
     | 99.99th=[ 1876]
   bw (  KiB/s): min=447400, max=640128, per=99.97%, avg=594167.11, stdev=23663.97, samples=119
   iops        : min=111850, max=160032, avg=148541.72, stdev=5915.99, samples=119
  lat (usec)   : 100=0.01%, 250=4.23%, 500=71.79%, 750=23.40%, 1000=0.18%
  lat (msec)   : 2=0.39%, 4=0.01%
  cpu          : usr=19.61%, sys=61.60%, ctx=150554, majf=0, minf=2
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=103.2%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=8915060,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=580MiB/s (609MB/s), 580MiB/s-580MiB/s (609MB/s-609MB/s), io=34.0GiB (36.5GB), run=60001-60001msec

Disk stats (read/write):
  vda: ios=9161062/58, merge=0/7, ticks=2747560/12, in_queue=2714548, util=99.87%