Initial NVME driver for DragonFly committed to master
Matthew Dillon
dillon at apollo.backplane.com
Sun Jun 5 12:08:42 PDT 2016
The need for a NVMe (SSD over PCIe) disk driver has slowly increased.
I finally decided to sit down and write one. After reviewing the drivers
in other OSs I decided to write ours from scratch.
This driver should be considered experimental, it has only been tested
on one NVMe card so far and I'm sure there are issues. We do not yet
have userland management support for it. It isn't finished but it
should be usable.
The driver is currently built as a module in master, so it must be
kldload'd to use (or put nvme_load="YES" in /boot/loader.conf).
And it is surprisingly quite fast. I built a ground-up SMP-friendly
implementation and even though it isn't using MSI-X yet, it's still
capable of some fairly significant bandwidth. So far I've only tested
it on a mini-PCIe card (2-lane or 4-lane, not sure) that I plugged into
an adapater and threw into a test box:
nvme0: <NVME-PCIe> port 0xe000-0xe0ff mem 0xf7e00000-0xf7e03fff
irq 16 at device 0.0 on pci1
nvme0: NVME Version 1.1 maxqe=16384 caps=00f000203c013fff
nvme0: Model SAMSUNG_MZVPV128HDGM-00000 BaseSerial S1XVNYAGA02988
nscount=1
nvme0: Request 18/6 queues, Returns 8/8 queues, nominal map
nvme0: Disk nvme0 ns=1 blksize=512 lbacnt=250069680 cap=119GB
serno=S1XVNYAGA02988-1
--
Concurrent random reads on the physical device using our
test/sysperf/randread.c with 512-byte reads yields a VERY
impressive 307,000 IOPS.
test40# randread /dev/nvme0s1b
tty nvme0 cpu
tin tout KB/t tps MB/s us ni sy in id
0 24 0.50 307806 150.30 5 0 51 10 34
0 24 0.50 307911 150.35 4 0 52 7 37
0 24 0.50 307793 150.29 5 0 55 8 32
0 24 0.50 307818 150.30 3 0 54 8 35
0 24 0.50 307893 150.34 4 0 53 8 35
--
Concurrent random reads through the HAMMER filesystem, freshly mounted
(uncached) is around 25000 IOPS @ 64KB, 1.5 GBytes/sec.
test40# randread /mnt/test1.dat
tty nvme0 cpu
tin tout KB/t tps MB/s us ni sy in id
0 12 64.00 24599 1537.43 0 0 36 5 59
0 23 63.99 24793 1549.40 1 0 37 5 57
0 23 64.00 24611 1538.10 1 0 43 3 53
--
Singe process sequential reads through the HAMMER filesystem,
freshly mounted (uncached), around 1.5 GBytes/sec.
test40# dd if=/mnt/test3.dat of=/dev/null bs=32k
tty nvme0 cpu
tin tout KB/t tps MB/s us ni sy in id
0 12 113.66 13537 1502.56 0 0 21 4 76
0 12 113.66 13103 1454.33 0 0 35 3 61
0 12 113.65 13070 1450.52 0 0 38 3 59
--
Concurrent sequential reads of two different files through the
HAMMER filesystem, freshly mounted (uncached), around
1.7 GBytes/sec.
test40# dd if=/mnt/test1.dat of=/dev/null bs=32k
test40# dd if=/mnt/test2.dat of=/dev/null bs=32k
tty nvme0 cpu
tin tout KB/t tps MB/s us ni sy in id
0 12 120.21 15066 1768.63 0 0 27 5 67
0 12 120.04 15002 1758.65 0 0 33 3 64
0 12 120.49 14930 1756.82 0 0 50 4 46
7516192768 bytes transferred in 7.931600 secs (947626318 bytes/sec)
7516192768 bytes transferred in 7.881519 secs (953647757 bytes/sec)
--
A single sequential read from the raw device, around 800MB/sec
using 32KB reads:
test40# dd if=/dev/nvme0s1b of=/dev/null bs=32k
tty nvme0 cpu
tin tout KB/t tps MB/s us ni sy in id
0 12 32.00 25863 808.23 0 0 5 2 92
0 12 32.00 25845 807.67 0 0 7 3 91
0 12 32.00 26407 825.23 0 0 5 3 92
17179869184 bytes transferred in 20.225951 secs (849397352 bytes/sec)
--
And finally, five concurrent sequential reads of the raw device
almost maxes out the device at 2.4 GBytes/sec:
dd if=/dev/nvme0s1b of=/dev/null bs=32k &
dd if=/dev/nvme0s1b of=/dev/null bs=32k &
dd if=/dev/nvme0s1b of=/dev/null bs=32k &
dd if=/dev/nvme0s1b of=/dev/null bs=32k &
dd if=/dev/nvme0s1b of=/dev/null bs=32k &
tty nvme0 cpu
tin tout KB/t tps MB/s us ni sy in id
1 27 32.00 76859 2401.84 1 0 21 4 74
0 12 32.00 76010 2375.33 0 0 22 4 74
0 12 32.00 76017 2375.53 0 0 23 4 73
0 12 32.00 75973 2374.16 1 0 20 4 75
So there we have it. A pretty good start for our new driver!
-Matt
Matthew Dillon
<dillon at backplane.com>
More information about the Users
mailing list