User:Strugglers/write-mostly

From BitFolk
Jump to navigation Jump to search

TL;DR

Wasn't able to detect any difference between using --write-mostly on an MD array and not using it. Performance of array was 1.65x higher than against single SM883, but lower than against single PM983.

The issues seems to be that despite --write-mostly being set, the reads are still being split fairly evenly between the devices instead of mostly at the PM983.

Setup

Host has 1x Samsung SM883 SATA SSD (/dev/sdc) and 1x Samsung PM983 NVMe (/dev/nvme0n1).

The PM983 has higher performance than the SM883. The hypothesis was that making a RAID array of the two of them and setting the SM883 to be "write mostly" would direct most reads to the PM983 which would result in higher read performance and little or no degradation in write performance.

A pair of fio tests were done on the following configurations:

  • ext4 filesystem directly on small partition on SM883
  • ext4 filesystem directly on small partition on PM983
  • ext4 filesystem on mdadm RAID10 (default settings) of SM883 and PM983
  • ext4 filesystem on mdadm RAID10 with SM883 member set to "write mostly"

The fio tests were:

  1. Random access read/write test against an 8GiB file, 75% of the IOs being reads.
  2. Random access read-only of an 8GiB file

In both cases:

  • All IOs 32kiB
  • Direct IO used (no OS cache)
  • Single process, IO depth of 32

Details

SM883

In the read/write test 12k read IOPS and ~4k write IOPS were achieved.

In the read-only test 16.5k IOPS were achieved. 

$ /opt/fio/bin/fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=limoncello_rw_ssd --filename=/mnt/fio --bs=32k --iodepth=32 --size=8G --readwrite=randrw --rwmixread=75
limoncello_rw_ssd: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=32
fio-3.13-42-g8066f
Starting 1 process
limoncello_rw_ssd: Laying out IO file (1 file / 8192MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=373MiB/s,w=127MiB/s][r=11.9k,w=4069 IOPS][eta 00m:00s]
limoncello_rw_ssd: (groupid=0, jobs=1): err= 0: pid=14037: Thu May 16 05:25:55 2019
  read: IOPS=12.0k, BW=376MiB/s (394MB/s)(6141MiB/16329msec)
   bw (  KiB/s): min=379328, max=390016, per=100.00%, avg=385081.91, stdev=2761.48, samples=32
   iops        : min=11854, max=12188, avg=12033.78, stdev=86.30, samples=32
  write: IOPS=4020, BW=126MiB/s (132MB/s)(2051MiB/16329msec); 0 zone resets
   bw (  KiB/s): min=123392, max=134528, per=100.00%, avg=128653.94, stdev=2519.98, samples=32
   iops        : min= 3856, max= 4204, avg=4020.41, stdev=78.75, samples=32
  cpu          : usr=4.44%, sys=23.33%, ctx=262033, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=196498,65646,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=376MiB/s (394MB/s), 376MiB/s-376MiB/s (394MB/s-394MB/s), io=6141MiB (6439MB), run=16329-16329msec
  WRITE: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=2051MiB (2151MB), run=16329-16329msec

Disk stats (read/write):
  sdc: ios=195155/65222, merge=0/66, ticks=396339/119750, in_queue=514412, util=99.47%

$ /opt/fio/bin/fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=limoncello_ro_ssd --filename=/mnt/fio --bs=32k --iodepth=32 --size=8G --readwrite=randread
limoncello_ro_ssd: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=32
fio-3.13-42-g8066f
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=518MiB/s][r=16.6k IOPS][eta 00m:00s]
limoncello_ro_ssd: (groupid=0, jobs=1): err= 0: pid=14070: Thu May 16 05:28:18 2019
  read: IOPS=16.5k, BW=517MiB/s (542MB/s)(8192MiB/15846msec)
   bw (  KiB/s): min=528448, max=530112, per=100.00%, avg=529384.19, stdev=566.62, samples=31
   iops        : min=16514, max=16566, avg=16543.23, stdev=17.74, samples=31
  cpu          : usr=4.39%, sys=23.87%, ctx=262110, majf=0, minf=264
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=517MiB/s (542MB/s), 517MiB/s-517MiB/s (542MB/s-542MB/s), io=8192MiB (8590MB), run=15846-15846msec

Disk stats (read/write):
  sdc: ios=258841/2, merge=0/1, ticks=497973/7, in_queue=495740, util=99.43%

PM983

In the read/write test 44.6k read IOPS and 14.9k write IOPS were achieved (roughly 3.7x SSD performance for both read and write).

In the read-only test 73.5k IOPS were achieved (almost 4.5x SSD). 

$ /opt/fio/bin/fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=limoncello_rw_nvme --filename=/mnt/fio --bs=32k --iodepth=32 --size=8G --readwrite=randrw --rwmixread=75
limoncello_rw_nvme: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=32
fio-3.13-42-g8066f
Starting 1 process
limoncello_rw_nvme: Laying out IO file (1 file / 8192MiB)
Jobs: 1 (f=1): [m(1)][60.0%][r=1388MiB/s,w=464MiB/s][r=44.4k,w=14.8k IOPS][eta 00m:02Jobs: 1 (f=1): [m(1)][80.0%][r=1390MiB/s,w=463MiB/s][r=44.5k,w=14.8k IOPS][eta 00m:01Jobs: 1 (f=1): [m(1)][100.0%][r=1393MiB/s,w=466MiB/s][r=44.6k,w=14.9k IOPS][eta 00m:00s]
limoncello_rw_nvme: (groupid=0, jobs=1): err= 0: pid=13846: Thu May 16 05:21:35 2019
  read: IOPS=44.6k, BW=1393MiB/s (1460MB/s)(6141MiB/4409msec)
   bw (  MiB/s): min= 1381, max= 1415, per=100.00%, avg=1394.31, stdev=10.46, samples=8
   iops        : min=44220, max=45282, avg=44618.00, stdev=334.72, samples=8
  write: IOPS=14.9k, BW=465MiB/s (488MB/s)(2051MiB/4409msec); 0 zone resets
   bw (  KiB/s): min=463488, max=483712, per=100.00%, avg=476560.00, stdev=7644.62, samples=8
   iops        : min=14484, max=15116, avg=14892.50, stdev=238.89, samples=8
  cpu          : usr=9.55%, sys=26.61%, ctx=155074, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=196498,65646,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=1393MiB/s (1460MB/s), 1393MiB/s-1393MiB/s (1460MB/s-1460MB/s), io=6141MiB (6439MB), run=4409-4409msec
  WRITE: bw=465MiB/s (488MB/s), 465MiB/s-465MiB/s (488MB/s-488MB/s), io=2051MiB (2151MB), run=4409-4409msec

Disk stats (read/write):
  nvme0n1: ios=190678/63724, merge=0/0, ticks=117973/17065, in_queue=135976, util=100.00%

$ /opt/fio/bin/fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=limoncello_ro_nvme --filename=/mnt/fio --bs=32k --iodepth=32 --size=8G --readwrite=randread
limoncello_ro_nvme: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=32
fio-3.13-42-g8066f
Starting 1 process
Jobs: 1 (f=1): [r(1)][-.-%][r=2293MiB/s][r=73.4k IOPS][eta 00m:00s]
limoncello_ro_nvme: (groupid=0, jobs=1): err= 0: pid=13879: Thu May 16 05:22:42 2019
  read: IOPS=73.5k, BW=2297MiB/s (2408MB/s)(8192MiB/3567msec)
   bw (  MiB/s): min= 2291, max= 2306, per=100.00%, avg=2297.62, stdev= 5.35, samples=7
   iops        : min=73318, max=73820, avg=73524.00, stdev=170.92, samples=7
  cpu          : usr=9.42%, sys=29.16%, ctx=148113, majf=0, minf=264
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=2297MiB/s (2408MB/s), 2297MiB/s-2297MiB/s (2408MB/s-2408MB/s), io=8192MiB (8590MB), run=3567-3567msec

Disk stats (read/write):
  nvme0n1: ios=249888/0, merge=0/0, ticks=107248/0, in_queue=108088, util=100.00%

mdadm RAID-10

In the read/write test 19.8k read IOPS and 6.6k write IOPS were achieved (roughly 1.65x SSD performance for both read and write). 55% of read IOs and 49% of write IOs were served by the PM983.

In the read-only test 34.4k IOPS were achieved (about 2.1x SSD). 67% of IOs were served by the PM983.

Array created like so: 

# mdadm --create \
        --verbose \
        --assume-clean \
        /dev/md4 \
        --level=10 \
        --raid-devices=2 \
        --size=10G \
        /dev/nvme0n1 /dev/sdc

At the moment that results in an n2 geometry with v1.2 metadata. 

$ /opt/fio/bin/fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=limoncello_rw_md --filename=/mnt/fio --bs=32k --iodepth=32 --size=8G --readwrite=randrw --rwmixread=75
limoncello_rw_md: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=32
fio-3.13-42-g8066f
Starting 1 process
limoncello_rw_md: Laying out IO file (1 file / 8192MiB)
Jobs: 1 (f=0): [f(1)][100.0%][r=621MiB/s,w=210MiB/s][r=19.9k,w=6728 IOPS][eta 00m:00s]
limoncello_rw_md: (groupid=0, jobs=1): err= 0: pid=13613: Thu May 16 05:15:21 2019
  read: IOPS=19.8k, BW=618MiB/s (648MB/s)(6141MiB/9931msec)
   bw (  KiB/s): min=610560, max=659840, per=99.89%, avg=632467.74, stdev=13363.39, samples=19
   iops        : min=19080, max=20620, avg=19764.58, stdev=417.67, samples=19
  write: IOPS=6610, BW=207MiB/s (217MB/s)(2051MiB/9931msec); 0 zone resets
   bw (  KiB/s): min=201216, max=216896, per=99.85%, avg=211208.53, stdev=4112.51, samples=19
   iops        : min= 6288, max= 6778, avg=6600.16, stdev=128.61, samples=19
  cpu          : usr=5.85%, sys=28.19%, ctx=195122, majf=0, minf=7
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=196498,65646,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=618MiB/s (648MB/s), 618MiB/s-618MiB/s (648MB/s-648MB/s), io=6141MiB (6439MB), run=9931-9931msec
  WRITE: bw=207MiB/s (217MB/s), 207MiB/s-207MiB/s (217MB/s-217MB/s), io=2051MiB (2151MB), run=9931-9931msec

Disk stats (read/write):
    md4: ios=191604/64020, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=98249/65649, aggrmerge=0/1, aggrticks=98789/59319, aggrin_queue=157768, aggrutil=98.40%
  nvme0n1: ios=108768/65649, merge=0/1, ticks=19172/1675, in_queue=20936, util=94.24%
  sdc: ios=87730/65650, merge=0/1, ticks=178407/116963, in_queue=294600, util=98.40%

$ /opt/fio/bin/fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=limoncello_ro_md --filename=/mnt/fio --bs=32k --iodepth=32 --size=8G --readwrite=randread
limoncello_ro_md: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=32
fio-3.13-42-g8066f
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=1077MiB/s][r=34.5k IOPS][eta 00m:00s]
limoncello_ro_md: (groupid=0, jobs=1): err= 0: pid=13650: Thu May 16 05:16:33 2019
  read: IOPS=34.4k, BW=1076MiB/s (1128MB/s)(8192MiB/7614msec)
   bw (  MiB/s): min= 1058, max= 1087, per=99.92%, avg=1075.08, stdev= 8.54, samples=15
   iops        : min=33874, max=34786, avg=34402.53, stdev=273.30, samples=15
  cpu          : usr=4.93%, sys=32.17%, ctx=177562, majf=0, minf=264
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=1076MiB/s (1128MB/s), 1076MiB/s-1076MiB/s (1128MB/s-1128MB/s), io=8192MiB (8590MB), run=7614-7614msec

Disk stats (read/write):
    md4: ios=254592/3, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=131072/4, aggrmerge=0/1, aggrticks=120583/7, aggrin_queue=120276, aggrutil=98.09%
  nvme0n1: ios=136114/4, merge=0/1, ticks=20934/1, in_queue=20108, util=97.06%
  sdc: ios=126030/5, merge=0/1, ticks=220232/14, in_queue=220444, util=98.09%

mdadm RAID-10 with --write-mostly

In the read/write test 19.7k read IOPS and ~6.6k write IOPS were achieved. Pretty much the same as without --write-mostly. 55% of read IOs and 50% of write IOs were served by the PM983. Again almost exactly the same as without --write-mostly

In the read-only test 34.3k IOPS were achieved (about 2.1x SSD). 52% of IOs were served by the PM983. Very slightly less than without --write-mostly.

Array created like so: 

# mdadm --create \
        --verbose \
        --assume-clean \
        /dev/md4 \
        --level=10 \
        --raid-devices=2 \
        --size=10G \
        /dev/nvme0n1 --write-mostly /dev/sdc

$ /opt/fio/bin/fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=limoncello_rw_md_writemostly --filename=/mnt/fio --bs=32k --iodepth=32 --size=8G --readwrite=randrw --rwmixread=75
limoncello_rw_md_writemostly: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=32
fio-3.13-42-g8066f
Starting 1 process
Jobs: 1 (f=1): [m(1)][100.0%][r=628MiB/s,w=212MiB/s][r=20.1k,w=6796 IOPS][eta 00m:00s]
limoncello_rw_md_writemostly: (groupid=0, jobs=1): err= 0: pid=13410: Thu May 16 05:09:00 2019
  read: IOPS=19.7k, BW=617MiB/s (647MB/s)(6141MiB/9953msec)
   bw (  KiB/s): min=592670, max=653184, per=99.87%, avg=630913.58, stdev=14060.91, samples=19
   iops        : min=18520, max=20412, avg=19716.11, stdev=439.43, samples=19
  write: IOPS=6595, BW=206MiB/s (216MB/s)(2051MiB/9953msec); 0 zone resets
   bw (  KiB/s): min=202411, max=218304, per=99.81%, avg=210666.68, stdev=4873.46, samples=19
   iops        : min= 6325, max= 6822, avg=6583.32, stdev=152.33, samples=19
  cpu          : usr=4.53%, sys=29.82%, ctx=194542, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%

     issued rwts: total=196498,65646,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=617MiB/s (647MB/s), 617MiB/s-617MiB/s (647MB/s-647MB/s), io=6141MiB (6439MB), run=9953-9953msec
  WRITE: bw=206MiB/s (216MB/s), 206MiB/s-206MiB/s (216MB/s-216MB/s), io=2051MiB (2151MB), run=9953-9953msec

Disk stats (read/write):
    md4: ios=195268/65241, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=98249/65649, aggrmerge=0/1, aggrticks=98906/59358, aggrin_queue=157720, aggrutil=98.38%
  nvme0n1: ios=107908/65649, merge=0/1, ticks=19970/1681, in_queue=21056, util=94.14%
  sdc: ios=88590/65650, merge=0/1, ticks=177842/117036, in_queue=294384, util=98.38%

$ /opt/fio/bin/fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=limoncello_ro_md_writemostly --filename=/mnt/fio --bs=32k --iodepth=32 --size=8G --readwrite=randread
limoncello_ro_md_writemostly: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=32
fio-3.13-42-g8066f
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=1072MiB/s][r=34.3k IOPS][eta 00m:00s]
limoncello_ro_md_writemostly: (groupid=0, jobs=1): err= 0: pid=13447: Thu May 16 05:10:52 2019
  read: IOPS=34.3k, BW=1073MiB/s (1125MB/s)(8192MiB/7638msec)
   bw (  MiB/s): min= 1058, max= 1079, per=99.94%, avg=1071.85, stdev= 7.14, samples=15
   iops        : min=33868, max=34550, avg=34299.07, stdev=228.40, samples=15
  cpu          : usr=4.82%, sys=31.83%, ctx=179195, majf=0, minf=263
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=1073MiB/s (1125MB/s), 1073MiB/s-1073MiB/s (1125MB/s-1125MB/s), io=8192MiB (8590MB), run=7638-7638msec

Disk stats (read/write):
    md4: ios=253777/3, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=131072/4, aggrmerge=0/1, aggrticks=120986/6, aggrin_queue=120462, aggrutil=98.22%
  nvme0n1: ios=135699/4, merge=0/1, ticks=21699/0, in_queue=21100, util=97.20%
  sdc: ios=126445/5, merge=0/1, ticks=220274/13, in_queue=219824, util=98.22%
Retrieved from "https://tools.bitfolk.com/w/index.php?title=User:Strugglers/write-mostly&oldid=1510"