$OpenBSD: nvme.4,v 1.2 2016/04/14 11:53:37 jmc Exp $
Copyright (c) 2016 David Gwynne <dlg (at] openbsd.org>
Permission to use, copy, modify, and distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
.Dd September 21, 2016 .Dt NVME 4 .Os .Sh NAME .Nm nvme .Nd Non-Volatile Memory Host Controller Interface .Sh SYNOPSIS .Cd "nvme* at pci? dev ? function ?" .Sh DESCRIPTION The .Nm driver provides support for NVMe, or NVM Express, storage controllers conforming to the Non-Volatile Memory Host Controller Interface specification. Controllers complying to specification version 1.1 and 1.2 are known to work. Other versions should too for normal operation with exception of some passthrough commands.
p Driver supports following features: l -bullet -compact -offset indent t controller and namespace configuration and management using .Xr nvmectl 1 t highly parallel I/O using per-CPU I/O queues t PCI MSI/MSI-X attachment, and INTx for legacy systems .El
p On systems supporting MSI/MSI-X, .Nm driver uses per-CPU IO queue pairs for lockless and highly parallelized I/O. Interrupt handlers are scheduled on distinct CPUs. Driver allocates as many interrupt vectors as available, up to number of CPUs + 1. MSI supports up to 32 interrupt vectors within the system, MSI-X can have up to 2k. Each I/O queue pair has separate command circular buffer. .Nm specification allows up to 64k commands per queue, driver currently allocates 1024 items per queue by default. Command submissions are done always on current CPU, command completion interrupt is handled on CPU according to I/O queue ID - first I/O queue on CPU0, second I/O queue on CPU1 etc. Admin queue command completion is not tied to any CPU, it's handled by any CPU. To keep lock contention to minimum, it's recommended to keep this assignment, even thought it is possible to reassign the interrupt handlers differently, using .Xr intrctl 1 . Driver also uses soft interrupts to process command completions, in order to increase total system I/O capacity and throughput.
p On systems without MSI, driver uses single HW interrupt handler, for both admin and standard I/O commands. Command submissions are done on current CPU, command completion interrupt is handled on any available CPU. This leads to some lock contention, especially on command ccbs. Also, command completion handling must be done within the HW interrupt handler. .Sh FILES l -tag -width /dev/nvmeX -compact t Pa /dev/nvme* nvme device special files used by .Xr nvmectl 1 . .El .Sh SEE ALSO .Xr intro 4 , .Xr ld 4 , .Xr pci 4 , .Xr nvmectl 1 , .Xr MAKEDEV 8 , .Xr intrctl 1 .Rs .%A NVM Express, Inc. .%T "NVM Express - scalable, efficient, and industry standard" .%D 2016-06-12 .%U http://nvmexpress.org/ .Re .Rs .%A NVM Express, Inc. .%T "NVM Express Revision 1.2.1" .%D 2016-06-05 .%U http://www.nvmexpress.org/wp-content/uploads/NVM_Express_1_2_1_Gold_20160603.pdf .Re .Sh HISTORY The .Nm driver first appeared in .Ox 6.0 and in .Nx 8.0 . .Sh AUTHORS .An -nosplit The .Nm driver was written by .An David Gwynne .Aq Mt dlg (at] openbsd.org for .Ox and ported to .Nx by .An NONAKA Kimihiro .Aq Mt nonaka (at] NetBSD.org . .An Jaromir Dolecek .Aq Mt jdolecek (at] NetBSD.org contributed to making this driver MPSAFE. .Sh NOTES At least some .Tn Intel .Nm adapter cards are known to require .Tn PCIe Generation 3 slot. Such cards do not even probe when plugged into older generation slot.
p Driver attaches and works fine also for emulated .Nm device under QEMU and .Tn Oracle .Tn VirtualBox . However, there seems to be some broken interaction between .Tn VirtualBox 5.1.6 and the driver, emulated .Nm controller responds to commands only first time it's attached, after reboot or module reload stops responding. Virtual machine must be completely powered off (or even killed) to fix.
p .Nm kernel module is currently only loadable for kernels configured with .Dq no ld .
Indexes created Sun Oct 26 17:10:13 GMT 2025