What Is NVMe Over TCP (NVMe/TCP)

Learn what is NVMe over TCP and how does NVMe over TCP work in an easy language.

NVMe Over TCP Explained

The increasing demand for cloud computing needs an efficient and affordable storage system. Though conventional hard disk drives (HDDs) are dirt cheap, these aren’t capable to host performance-driven software or computational processes on the cloud.

Hence, to address this issue in mass storage for computer devices, NVM Express Inc. published the architecture for NVMe SSD, a blazing-fast internal storage device for computers over PCIe or Peripheral Component Interconnect Express bus.

However, PCIe is only for direct data communications between the RAM and NVMe SSD. How would one implement the same for a cloud computing environment where host and client computers are located in remote places? These devices can only communicate via the internet or intranet.

To address this challenge, NVMe over Fabric or NVMe-oF protocol was designed to allow NVMe network storage solution development. It uses the NVMe logical-device specification interface to transfer data between the host computer or server and a storage area network (SAN) or direct attached storage (DAS).

But, NVMe-oF deployment is highly expensive even for large enterprises. Also, as NVMe-oF mainly demands fiber optics cable as the transport medium of data from point A to point B, it’s not just feasible to deploy NVMe-oF for public clouds.

To address this issue, NVM Express Inc. has come up with the NVMe over TCP or NVMe/TCP protocol.

What Is NVMe Over TCP (NVMe/TCP)?

NVMe/TCP is an extended edition of the NVMe-based storage I/O command specification. It’s the advanced version of NVMe-oF and defines the functionalities of the NVMe data transfer protocol to the message-based communication protocol, TCP.

There is an NVMe/TCP Transport Specification that describes the followings:

• How to map NVMe queues following a standard set of rules
• How to create capsules for NVMe over Fabric
• How to deliver NVMe-oF capsules to fiber, InfiniBand, and non-Fiber network transport systems using the familiar Transport Control Protocol or TCP

In a nutshell, the combination of NVMe and TCP, NVMe/TCP transfers data and commands faster and in an error-free way between NVMe-oF hosts and NVMe-oF controller devices.

Such devices can be connected over Internet and Intranet using an Ethernet-based TCP/IP network. The connectivity medium could be optical fiber cables or conventional Cat 5 and Cat 6 cables

Why NVMe Over TCP?

The original NVMe transport system was intended for SSDs that’ll go directly into the PC system, mainly on the motherboard. The transport protocol has been initially optimized for NVM PCIe SSDs that connect to the servers or workstations internally.

Later on for blazing fast and massive data storage requirements on the cloud, the NVMe transport protocol hotwired with transport binding specifications that cloud data centers mostly use, like Fibre Channel, RDMA over Converged Ethernet (RoCE), Internet Wide Area RDMA Protocol (iWARP), and Infiniband.

Though the above NVMe modification works just fine, it’s not suitable for public clouds since specialized hardware equipment is needed from the client end. Hence, TCP transport binding system was incorporated which is readily available globally and reliable over long-distance networks.

The Core Components of NVMe Over TCP

An NVMe/TCP transport specification for NVMe-oF includes the following network components:

• The remote data access system using the NVMe protocol follows the Open Systems Interconnection (OSI) Model for computer network systems.
• NVMe/TCP utilizes the Ethernet networking technology that operates in the OSI physical and OSI datalink layer.
• For communication protocols and packet data exchange policies, NVMe/TCP relies on TCP/IP protocol.
• NVMe command sets to convert all user-issued commands and application I/O into a specific NVMe command table like admin, I/O, fabric, etc.
• The remote transfer of NVMe commands, in a message-based fabric, takes place using capsules. These capsules could be response or command capsules. NVMe command capsules represent a unit of NVMe data exchange.
• A subsystem and host controller communicate between themselves by exchanging NVMe/TCP Protocol Data Units or NVMe/TCP PDUs.
• There is a three-step connection process between the controller subsystem and the host. These steps or phases are NVMe-oF Connect, NVMe/TCP Connect, and Fabric Connect.
• In an NVMe/TCP transport binding system, dynamic I/O controllers are automatically created and assigned to the host computer when a subsystem connects successfully with the host NVMe-OF.

Benefits of NVMe Over TCP

• TCP is the most widely used Internet or Intranet data transfer protocol and is available with every data center and the clients of cloud computing resources.
• The data centers and their clients don’t need to replace existing network hardware like routers, switches, and NICs.
• TCP can use both fiber cable and Cat 5 or Cat 6 cable and hence maintenance is quite easy.
• TCP transport binding for NVMe-oF offers high performance and low latency. Thus it’s perfectly suited for mission-critical and CPU-intensive processes on the host computer and easily accessed by multiple remote clients without any read/write conflicts.
• TCP offers dynamic routing. Thus, when compared with remote direct memory access (RDMA), TCP is the highest performer in large-scale data center and cloud operations from longer distances.
• TCP has a larger development community than any other NVMe transport specifications.

Challenges of NVMe Over TCP

• When using a TCP stack, you must allow the CPU to resolve most of the TCP operations like calculating checksums. Thus, on a larger scale, when multiple client workstations will request to write and read data from networked storage devices through the host PC, the load on the host CPU will increase.
• Hyperconverged infrastructure must be simple. However, when you use NVMe/TCP, you make it complex.
• NVMe over TCP also has some security weaknesses that hackers can utilize to infiltrate your cloud operations, steal the data, and make confidential data available on the dark web unless you deploy a stringent cybersecurity and antivirus solution.

NVMe Over TCP: Final Words

TCP is the leading protocol across public or private networked data centers. Hence, NVMe/TCP is the first choice for providing affordable storage solutions over the internet, intranet, and cloud infrastructure.

This protocol is also scalable and reliable over long-distance network connections. Since the cloud storage read/write latency is equivalent to most fiber channel protocols, NVMe over TCP is the preferred cloud storage transport solution.

Next up, 3 alternatives to cloud storage