Optimizing NVME-Storage Systems With IO_URING

For a long time, storage devices were much slower than CPUs. A hard disk had to physically move a read/write head and wait for the correct sector to rotate underneath it. That mechanical delay dominated everything else, so software did not need an especially efficient way to keep the device busy.

SSDs removed the moving parts. SATA SSDs became much faster, but requests from multiple CPU cores still pass through one shared command queue. The cores must coordinate through this common queue, which limits how much independent I/O they can issue in parallel.

SATA requests from CPU cores pass through one shared command queue

NVMe was designed for parallel solid-state storage. Instead of one shared queue, it supports multiple submission and completion queue pairs. Different CPU cores or worker threads can use separate queues and keep many independent requests in flight. The SSD controller then has more work to perform and can execute requests concurrently across its internal flash resources.

That is the logical reason why NVMe scales better; fewer threads contend for one shared queue, more I/O latency can overlap, and the device sees enough independent work to use its internal parallelism. This queue model is what lets software feed that hardware efficiently.

NVMe lets CPU cores use multiple independent queue pairs

This hardware model changes how storage systems should issue I/O.

A storage system can keep more I/O in flight by using more threads. When one thread blocks on a read, another thread can submit the next request. This can improve throughput, but it ties I/O concurrency to the number of threads. As the thread count grows, the system spends more memory on thread stacks and more CPU time scheduling threads, switching between them, and coordinating shared state.

Linux already had asynchronous I/O through libaio. It avoids blocking one application thread for every request, but it was designed around a narrower set of file I/O operations. It did not provide one common asynchronous interface for the wider mix of operations used by a storage system.

Libraries like SPDK takes a more specialized path. It maps an NVMe device into userspace and lets the storage engine submit commands directly. This removes much of the kernel I/O path, but the engine must be designed around SPDK's polling and userspace-driver model. It also gives up the normal file abstraction and ties the design closely to supported hardware.

What storage systems needed was a general asynchronous Linux interface that keeps the kernel's file and device abstractions. It should allow many requests to remain in flight without making the application create one blocking thread for each request. This is the problem io_uring addresses.

In this post, we will understand io_uring, then build the simplest page-read path and improve it with batched submissions, registered resources, submission polling, and specialized NVMe options.

Understanding IO_Uring Design

The io_uring is a Linux asynchronous I/O interface introduced in Linux 5.1. It is built around two ring buffers shared between userspace and the kernel:

• Submission Queue (SQ): The application places I/O requests into this queue.

• Completion Queue (CQ): The kernel places the result of completed I/O requests into this queue.

Each submission request entry in the SQ is called Submission Queue Entry, or SQE. Each completion request entry in the CQ is called Completion Queue Entry, or CQE.

Since the Submission Queue and Completion Queue are shared between userspace and the kernel, adding an SQE and consuming CQE that is already ready do not require a system call. This significantly improve the performance for critical storage systems.

By default, after submitting SQEs, the application must notify the kernel to process them, which requires system call. The io_uring provides ways to optimize it, and we will see this in this post.

Unlike the thread-based approach, io_uring does not require the application to dedicate one blocking thread to every in-flight request. The application can submit I/O, continue running other work, and return when a CQE is ready. The same interface works with regular files, block devices, and network sockets, so the storage system is not tied to a userspace NVMe driver.

io_uring is Linux-specific, so it is not portable across operating systems. Within Linux, however, it provides one asynchronous interface over the kernel's existing I/O abstractions.

Following diagram illustrate the working on io_uring.

High-level io_uring submission and completion flow

At a high level, an application thread prepares a request and submits its SQE through the Submission Queue. The kernel consumes the SQE and performs the operation. When the operation finishes, the kernel publishes a CQE in the Completion Queue, and the application thread consumes the result.

Submitting the SQE does not require the thread to wait until the I/O finishes. While the request is in flight, the thread can prepare more I/O operations, fill additional SQEs, submit another batch, or drain completions from earlier operations.

Reading Disk Pages With IO_Uring

Consider a storage system that has located several pages on disk but cannot find them in memory. It already knows each page's file offset and has reserved an output buffer where its result will be read. It now needs to submit those page read-requests using io_uring, and finally read the completed results from it. Following is the diagram illustrating it.

A buffer-pool page miss handled through io_uring

Let's begin with a straightforward implementation. For each page, we prepare an SQE containing the file-descriptor, offset, page-size, and destination buffer. We store the page identifier in user_data, which helps identify each page-id in the completion queue, and finally submit the requests.

for (size_t i = 0; i < page_ids.size(); ++i) {
    io_uring_sqe* sqe = io_uring_get_sqe(&ring);

    io_uring_prep_read(
        sqe, fd, buffers[i], page_size, offsets[i]);
    
    // User data.
    io_uring_sqe_set_data64(sqe, page_ids[i]);

    io_uring_submit(&ring);
}

The loop does not wait for a page to finish before submitting the next one, so several reads can be in flight at the same time. However, each iteration calls io_uring_submit() and therefore enters the kernel once for every page.

After submitting the reads, we drain their completions in a separate loop:

for (size_t i = 0; i < page_ids.size(); ++i) {
    io_uring_cqe* cqe = nullptr;
    io_uring_wait_cqe(&ring, &cqe);

    // User data.
    auto page_id = io_uring_cqe_get_data64(cqe);

    complete_page_read(page_id, cqe->res);
    io_uring_cqe_seen(&ring, cqe);
}

Completions may arrive in a different order from submissions, so the value stored in user_data tells the storage system which page has finished. Once the read succeeds, the storage system can make the page available and continue its work.

Optimizing Page Reads

Our initial implementation provides a decent solution, but it is still not acceptable for a high-performance storage systems. In the following section we will explore ways to optimize our solutions.

Batch Submissions

The current design overlaps page reads, but each call to io_uring_submit() still crosses the userspace-kernel boundary. With the default submission mode, this means one system call for every page-read request. At a high request rate, the accumulated system-call overhead consumes CPU time and can increase latency.

Instead of submitting requests one at a time, we submit them in batches. A single system call can submit multiple prepared SQEs. This amortizes the system-call overhead across the batch while keeping multiple page reads in flight.

for (size_t i = 0; i < page_ids.size(); ++i) {
    io_uring_sqe* sqe = io_uring_get_sqe(&ring);

    io_uring_prep_read(
        sqe, fd, buffers[i], page_size, offsets[i]);
    io_uring_sqe_set_data64(sqe, page_ids[i]);
}

// Batch submit.
io_uring_submit(&ring);

Polling Under Sustained I/O

Even after batching, the storage system makes one system call for every batch. Under sustained I/O, it may generate a continuous stream of batches. At high request rates, the resulting system calls can consume significant CPU time.

SQPOLL avoids these submission system calls. A kernel thread keeps checking the Submission Queue and picks up new requests as worker threads add them.

Completions have a similar problem. NVMe normally sends an interrupt when a read completes. Under sustained I/O, handling a large number of interrupts can add CPU overhead and latency.

IOPOLL avoids these interrupts by actively checking supported direct-I/O storage for completed requests. This can reduce latency, but it keeps a CPU busy. For occasional reads, interrupt-driven completions are usually more efficient. Below is the snippet for the same.

io_uring_params params{};
params.flags =
    IORING_SETUP_SQPOLL |
    IORING_SETUP_IOPOLL;

io_uring ring{};
io_uring_queue_init_params(
    queue_depth, &ring, &params);

int fd = open(path, O_RDONLY | O_DIRECT);

// Prepare the batched SQEs as before.
io_uring_submit(&ring);

Register Files and Buffers

Batching reduces submission overhead, but it does not remove the work performed for each SQE. Every read still carries a file descriptor and a userspace buffer address. Before starting the I/O, the kernel must resolve the descriptor to a file and establish safe access to the buffer.

A storage system keeps its data files open for a long time and repeatedly reads pages into the same set of buffers. Repeating the file lookup and buffer setup for every page adds CPU cycles without sending any additional work to the SSD.

The io_uring lets us mitigate this setup by registering files and buffers when the ring is initialized. The kernel can then reuse those file references and buffers for later page reads instead of repeating the same setup for every SQE.

std::vector<iovec> registered_buffers(buffers.size());

for (size_t i = 0; i < buffers.size(); ++i) {
    registered_buffers[i] = {buffers[i], page_size};
}

int rc = io_uring_register_buffers(
    &ring, registered_buffers.data(), registered_buffers.size());

if (rc < 0) {
    handle_error(rc);
}

for (size_t i = 0; i < page_ids.size(); ++i) {
    io_uring_sqe* sqe = io_uring_get_sqe(&ring);

    io_uring_prep_read_fixed(
        sqe,
        fd,
        buffers[i],
        page_size,
        offsets[i],
        static_cast<int>(i));

    io_uring_sqe_set_data64(sqe, page_ids[i]);
}

io_uring_submit(&ring);

Registration is not free. Registered buffers pin memory, which prevents that memory from being reclaimed while it remains registered. A storage system should therefore register a deliberately sized I/O pool.

NVMe Passthrough

A normal read passes through the kernel's generic file and I/O layers before it reaches the NVMe device. These layers provide useful abstractions, but also add overhead around every request.

A storage engine that manages a raw NVMe device may want to avoid part of this generic path and send NVMe commands directly through the device driver. This is known as NVMe passthrough. The io_uring supports this with IORING_OP_URING_CMD.

io_uring ring{};

// Initialization omitted.

io_uring_sqe* sqe = io_uring_get_sqe(&ring);
sqe->opcode = IORING_OP_URING_CMD;
sqe->fd = nvme_fd;
sqe->cmd_op = NVME_URING_CMD_IO;

auto* cmd = reinterpret_cast<nvme_uring_cmd*>(sqe->cmd);

std::memset(cmd, 0, sizeof(*cmd));

cmd->opcode = 0x02; // NVMe Read
cmd->nsid = namespace_id;
cmd->addr = reinterpret_cast<uint64_t>(buffer);
cmd->data_len = page_size;

// More initialization omitted.

io_uring_submit(&ring);

Passthrough can reduce work in the generic I/O path, but it is a specialized option. The storage engine becomes tied to Linux and NVMe and must manage the device more directly.

Bringing It All Together

Let's come back to where we started. NVMe is designed to process many requests in parallel, while io_uring gives a storage system an efficient way to keep those requests in flight.

The page-read example showed how to use io_uring. Our first version submitted every read separately. Moving the submission outside the loop allowed one system call to submit the complete batch, reducing CPU overhead while keeping enough work available for the NVMe device.

Registered resources removed repeated setup, while polling reduced system-call or interrupt overhead for sustained I/O. NVMe passthrough remains useful only when a storage system needs direct control of the device.

I hope this blog post made the relationship between NVMe, io_uring, and storage systems tangible. See you next time, and happy learning :-).