Direct NVMe Access vs. Virtual Storage Arrays: Real‑World IOPS Comparison

In today’s data‑intensive world, the race for faster storage is relentless. Enterprises and developers are torn between two seemingly opposite approaches: mounting raw NVMe devices directly on a server for blistering I/O, or leveraging the flexibility of cloud‑based storage‑area networks (SANs) that abstract physical media behind a virtual layer. Both promise high performance, but the real question is—how do they compare when you measure the ultimate metric that matters to most workloads: I/O operations per second (IOPS)? This article breaks down the technical nuances, benchmarks, and practical considerations of direct NVMe access versus virtual storage arrays, giving you the insight needed to make an informed architecture decision.

1. Understanding the Fundamentals

1.1 Direct NVMe Access

NVMe (Non‑Volatile Memory Express) is a protocol designed from the ground up for solid‑state storage. When a server attaches an NVMe drive via PCIe, the operating system can issue commands directly to the device with minimal software overhead. The result is:

• Ultra‑low latency (often < 50 µs)
• Massive parallelism (up to 64 K queues, each with 64 K commands)
• Performance that scales linearly with the number of PCIe lanes

1.2 Virtual Storage Arrays (SANs) in the Cloud

Cloud providers such as AWS, Azure, and Google Cloud expose block storage through SAN‑style services (EBS, Azure Managed Disks, Persistent Disks). Under the hood, these services aggregate hundreds or thousands of physical disks—often a mix of NVMe, SSD, and HDD—into a logical pool. The pool is presented to the VM over a network protocol (iSCSI, NVMe‑oF, Fibre Channel over Ethernet). Key characteristics include:

• Elastic capacity and on‑demand provisioning
• Built‑in replication, snapshots, and multi‑zone durability
• Performance guaranteed via provisioned IOPS or throughput caps

2. The IOPS Benchmark Landscape

2.1 How IOPS Is Measured

IOPS is a function of three variables: block size, read/write ratio, and latency. A typical benchmark (e.g., FIO) runs a mix of 4 KB random reads/writes because this size stresses the storage controller and mirrors many database and transactional workloads. For a fair comparison, both environments should use the same queue depth and concurrency settings.

2.2 Raw NVMe Numbers

Modern enterprise NVMe drives (e.g., Intel Optane PMEM, Samsung PM1733) routinely deliver:

1. Read IOPS: 500 k – 1 M for 4 KB random reads
2. Write IOPS: 300 k – 800 k for 4 KB random writes
3. Latency: 30 – 50 µs read, 45 – 70 µs write

When multiple NVMe devices are striped (RAID‑0 or software‑defined), the aggregate IOPS can exceed 2 M, limited mainly by CPU generation capacity and PCIe bandwidth.

2.3 Cloud SAN IOPS Profiles

Cloud SANs typically expose two performance models:

• Provisioned IOPS: You specify a target (e.g., 10 k, 30 k, 100 k IOPS). The service guarantees that level, subject to burst credits.
• Throughput‑Optimized: You specify MB/s; IOPS is derived from the chosen block size.

Real‑world tests on leading providers show:

Provider	Max Provisioned IOPS (4 KB)	Typical Latency
AWS EBS io2	64 k per volume (up to 256 k with multi‑attach)	≈ 150 µs
Azure Ultra Disk	160 k per disk (aggregated higher)	≈ 120 µs
Google Cloud PD‑SSD	120 k per disk	≈ 130 µs

Even the highest‑end cloud disks fall short of raw NVMe numbers, primarily because of network‑stack overhead and the shared nature of the underlying hardware.

3. Where the Gaps Appear—and Why

3.1 Latency Overheads

Direct NVMe bypasses the network stack entirely. In contrast, a cloud SAN adds at least one round‑trip across the data‑center fabric (often 10 GbE or 25 GbE). Even with NVMe‑over‑Fabric (NVMe‑oF), you still incur:

• TCP/UDP processing or RDMA context switches
• Virtualization overhead (hypervisor, vSwitch)
• Potential queueing in shared back‑end storage controllers

The cumulative effect pushes latency into the 100‑200 µs range, which directly throttles IOPS for small block sizes.

3.2 Contention and Multi‑Tenant Noise

Cloud storage pools are multi‑tenant. While providers isolate workloads, “noisy neighbor” effects can still surface during peak periods, causing temporary drops in IOPS. Direct NVMe, being a dedicated device, eliminates this variable.

3.3 Scaling Considerations

If you need > 1 M IOPS, a single NVMe drive won’t suffice; you must stripe multiple devices and ensure the CPU, memory, and PCIe lanes keep up. In the cloud, you can attach many volumes to a single VM, but each attachment incurs its own network path and may hit aggregate limits (e.g., total IOPS per instance type). Some providers offer elastic SANs that automatically scale, yet the scaling is still bounded by the underlying fabric.

3.4 Cost vs. Performance Trade‑offs

Raw NVMe hardware is a capital expense (CAPEX) with a predictable depreciation curve. Cloud SANs are an operational expense (OPEX) and charge per provisioned IOPS or per GB‑month. For workloads that truly need millions of IOPS, the per‑IOP cost in the cloud can become prohibitive, whereas on‑prem NVMe can be amortized over years.

4. Choosing the Right Approach for Your Workload

4.1 Latency‑Critical Applications

High‑frequency trading, real‑time analytics, and ultra‑low‑latency databases (e.g., FaunaDB, VoltDB) benefit most from direct NVMe. Sub‑50 µs latency translates into faster transaction commits and tighter SLAs.

4.2 Elastic, Variable‑Load Scenarios

Web‑scale SaaS platforms, CI/CD pipelines, and big‑data processing often experience bursty I/O patterns. The elasticity of cloud SANs—instant provisioning, easy snapshotting, and cross‑zone replication—outweighs the raw IOPS advantage of NVMe.

4.3 Hybrid Strategies

Many enterprises adopt a hybrid model: critical hot data resides on on‑prem NVMe arrays, while bulk or archival data lives in the cloud. Technologies like NVMe‑oF gateways or storage‑class‑memory caching can bridge the gap, offering near‑NVMe latency for a subset of data while retaining cloud flexibility.

Conclusion

When you strip away abstractions, direct NVMe access delivers unmistakably higher IOPS and lower latency than any current cloud‑based SAN offering. The difference is most pronounced for small‑block, random workloads where every microsecond counts.

However, raw performance is only one piece of the puzzle. Cloud storage brings elasticity, built‑in durability, and a pay‑as‑you‑go model that can dramatically reduce operational complexity.

Ultimately, the decision hinges on your workload’s tolerance for latency, its scalability needs, and your organization’s cost structure. For ultra‑low‑latency, mission‑critical services, a dedicated NVMe deployment remains the gold standard. For workloads that value flexibility, global availability, and simplified management, a virtual storage array—especially one that supports NVMe‑over‑Fabric—offers a compelling, if slightly slower, alternative. By understanding the IOPS trade‑offs outlined above, you can architect a storage solution that aligns performance with business objectives, whether that means leaning fully on the edge of the data center or embracing the cloud’s ever‑expanding horizon.