With the rapid advancement of technology, numerous industry applications place demanding requirements on network performance and isolation. For instance:
- Low latency: Some futures companies prefer to host their servers in data centers and independently develop trading programs to achieve rapid response (in microseconds) to market fluctuations. Particularly for high-frequency trading, even milliseconds or microseconds of latency can significantly impact the final transaction outcomes. In such scenarios, reducing network latency can substantially enhance business value.
- High bandwidth: High-performance computing (HPC) scenarios, such as scientific research computing, simulation, and emulation, often require the transmission and processing of large-scale data through parallel computing clusters comprising hundreds or even thousands of computing nodes. These nodes necessitate frequent communication to synchronize computational results, exchange data, and coordinate computational tasks. Consequently, high network bandwidth is crucial to enhancing data transmission efficiency and overall computing performance.
Network I/O virtualization is an effective approach that optimizes network resource utilization and enhances I/O performance. Currently, there are three mainstream implementations: emulation, para-virtualization, and pass-through. To facilitate users in leveraging network virtualization, SmartX has introduced PCI NIC pass-through capability in SMTX OS 5.1. This feature, combined with the existing SR-IOV pass-through and virtual NIC capabilities supported in previous versions, enables SMTX OS 5.1 to comprehensively support ELF (SmartX native hypervisor) users in harnessing the benefits of network I/O virtualization. Users can select the most suitable options based on their requirements for network performance, network isolation, and cost efficiency in various use cases, including futures trading, HPC, and more.
In this article, we will introduce SmartX HCI’s network I/O virtualization capabilities and test the performance of NICs under PCI pass-through and SR-IOV pass-through modes.
Network I/O Virtualization Features in SMTX OS 5.1
Features
Currently, SMTX OS (with ELF) supports three types of network devices: virtual NIC, SR-IOV pass-through NIC, and PCI pass-through NIC.
- Virtual NIC: Using software to simulate the physical NIC, allowing VMs to interconnect with the external network.
- PCI pass-through: Passing the NIC on the host to a VM through the PCI pass-through. This NIC is exclusively used by the VM.
- SR-IOV pass-through: A physical NIC that supports SR-IOV is virtualized into multiple VFs (Virtual Functions) and directly mounted to VMs as SR-IOV pass-through NICs. Multiple VMs can share the same physical NIC for communication.
Each host can use different NIC models*, and each VM can be mounted with multiple network devices.
*See the appendix for the list of compatible devices.
Virtual NIC
Virtual NICs are the most widely used network adapters in virtualization. In addition to connecting VMs with the network, virtual NICs also isolate network traffic between VMs through network isolation policies. Combined with virtual switches (VDS) and VM networks, virtual NICs can help create complex virtual network topologies.
Virtual NICs provide excellent flexibility and elasticity. The configuration of virtual NIC on a VM can be modified as needed. Users can also accelerate the deployment of new business services by cloning the existing configuration of virtual NIC. Moreover, since the software and hardware are decoupled, VMs can retain the configuration of the virtual NIC during snapshot and migration operations, making it easy to rebuild VMs on another host.
Although a virtual NIC typically offers satisfactory performance for most use cases, the emulation of a physical NIC (used by the E1000 vNIC type) or para-virtualization (like the VIRTIO model NIC) can introduce additional performance overhead. Moreover, when sharing VDS and VM networks, certain business VMs demanding high network performance might generate large traffic flows, potentially monopolizing network resources and causing an imbalance in network resource allocation among other VMs.
PCI pass-through NIC
PCIe pass-through technology allows VMs to directly access and use the NIC on the SMTX OS host. The passed-through NIC is compatible with most guest OS and PCIe-bus-compliant NICs. By bypassing the virtualization layer and directly utilizing the physical NIC, the VM operating system shortens the data transmission path, enabling VMs to obtain performance and features close to that of the physical server using the physical NIC. Furthermore, the VM’s exclusive ownership of a physical NIC can enhance network isolation and ensure security between VMs.
However, with PCI pass-through, a physical NIC cannot be shared simultaneously by multiple VMs. As a result, users must configure multiple physical NICs to fulfill the requirements of multiple VMs. In addition, VMs with PCI pass-through NIC attached do not support operations such as HA and hot migration.
SR-IOV pass-through NIC
SR-IOV (Single Root – I/O Virtualization) is a hardware-based virtualization solution in which a physical NIC can be split into multiple VFs (Virtual Functions) and mounted to VMs as SR-IOV pass-through NICs. This allows multiple VMs to share a single physical NIC, which improves performance while saving costs. Users can assign multiple SR-IOV pass-through NICs to the same VM according to their needs.
*To use this feature, the physical NIC should support SR-IOV. Certain NICs also require the installation of the appropriate driver in OS.
Use Cases
| Solutions | Virtual NIC | PCI Pass-through NIC | SR-IOV Pass-through NIC |
| Assessment |
|
|
|
| Common Application Scenarios |
|
|
|
| Advantages & Value |
|
|
|
Performance Test
To demonstrate SMTX OS PCI pass-through and SR-IOV pass-through capabilities, we conducted performance tests on different NICs.
PCI Pass-through
We tested two NICs’ performance (Solarflare and Mellanox) in three scenarios through netperf and iperf3 tools. In particular, we compared the I/O throughput and latency of VMs that enabled PCI pass-through under VM-VM and VM-host scenarios with that under the host-host scenario.
*See Appendix for hardware configuration and test tools.
Results
When a physical NIC is passed through to a VM, the throughput of the NIC under the TCP decreases slightly. This is because the VM needs to share physical resources with other VMs and applications on the host, which typically requires additional computing resources for KVM virtualization to manage and schedule resources and thus increases overhead and network latency.
The throughput of the NIC under the TCP also decreases slightly. This is because the network stack has to bypass the virtualization layer to communicate directly with the physical network adapter, and using PCI pass-through network devices can introduce additional overhead, such as interrupt handling and memory access.
The latency of the NIC increases slightly when a physical NIC is passed through to a VM. This is because the VM’s network stack has to communicate directly with the physical network adapter bypassing the virtualization layer, and using PCI pass-through network devices can introduce additional overhead such as interrupt handling and memory access. The impact of this overhead is even more pronounced in TCP traffic, which tends to involve more frequent and smaller packets than UDP traffic.
Conclusions
After enabling PCI pass-through mode, VMs in SmartX HCI can directly access the physical NIC. But this also introduces additional overhead, such as DMA programming, interrupt handling, etc., which consume additional CPU and memory resources. Moreover, multiple layers of processing and forwarding in the virtualization may also affect network throughput.
However, the additional overhead brought by PCI pass-through is negligible. Overall, PCI pass-through NIC’s performance is very close to that of physical NIC, meaning that this feature can fully meet the requirements of low latency and high performance for VM networks.
SR-IOV Pass-through
We tested two NICs’ performance (Solarflare and Mellanox) through sfnettest tool. In particular, we compared the latency of VMs that mounted SR-IOV pass-through NIC under VM-VM and VM-host scenarios with that under the host-host and VM-VM (mounted VIRTIO NIC) scenarios.
For Solarflare and Mellanox NICs, we also tested their performance after enabling the respective acceleration schemes.
*See Appendix for hardware configuration and test tools.
Results
Under cross-switch connections, compared with the average latency of 14.4 us for host-host scenario (benchmark), the SR-IOV pass-through Mellanox NIC can provide performance close to that of a physical NIC in both VM-VM and VM-host scenarios without enabling the acceleration scheme. After enabling the acceleration scheme, the latency can be reduced to 2-3 us in both scenarios.
In the direct connection mode, compared to the average latency of 63 us for VM-VM using a VIRTIO NIC, the SR-IOV pass-through Solarflare NIC can significantly reduce the latency in both VM-VM and VM-host scenarios without enabling the acceleration scheme. After enabling the acceleration scheme, the latency can be reduced to 2-3 us in both scenarios.
*The size of a futures trading order is usually between 64 and 128 bytes. The average delay (i.e., mean) measured in this test is also calculated based on the packet size of 64 bytes (i.e., size = 64).
Conclusions
After enabling SR-IOV pass-through mode, VMs in SmartX HCI can directly access VFs virtualized through physical NICs. Overall, VMs using SR-IOV pass-through NICs can significantly reduce latency compared to using VIRTIO NICs and achieve performance that is very close to that of the physical host. After enabling the acceleration scheme, the average latency of Solarflare and Mellanox NICs can be further reduced.
Besides network I/O virtualization capabilities, SmartX HCI 5.1 also improves performance and reliability through updated features such as GPU Pass-through and vGPU support, innovated DRS implementation, and temporary replica strategy. For more information, please refer to our previous articles:
Introducing SmartX HCI 5.1, Full Stack HCI for Both Virtualized and Containerized Apps in Production
GPU Passthrough & vGPU: Using GPU Application in Virtualization with SMTX OS 5.1
Improving Resource Utilization: Innovative Implementation of DRS in SmartX HCI
Appendix
SMTX OS 5.1 NIC Compatible List
| Brands | NIC Models | Support PCI Pass-through | Support SR-IOV Pass-through |
| Solarflare | Solarflare XtremeScale X2522-25G Adapter | ✓ | ✓ |
| Solarflare Flareon Ultra 8000 Series 10G Adapter | ✓ | ||
| Mellanox | Mellanox Technologies MT27710 Family [ConnectX-4 Lx] | ✓ | ✓ |
| Mellanox Technologies MT27800 Family [ConnectX-5] | ✓ | ✓ | |
| Mellanox Technologies MT2894 Family [ConnectX-6 Lx] | ✓ | ||
| Intel | Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02) | ✓ | |
| Intel Corporation Ethernet Connection X722 for 10GbE SFP+ (rev 04) | ✓ | ✓ | |
| Intel Corporation I350 Gigabit Network Connection (rev 01) | ✓ |
*We only display NICs that have been testified in supporting PCI pass-through.
PCI Pass-through Test
Hardware Configuration
| Hardware | Configuration |
| CPU (Host A) | Intel(R) Xeon(R) Gold 6238 CPU @ 2.10GHz 88 CPUs |
| NIC (Host A) | Solarflare Communications Device 0b03 |
| CPU (Host B) | Intel(R) Xeon(R) Gold 6238 CPU @ 2.10GHz 88 CPUs |
| NIC (Host B) | Mellanox Technologies MT27800 Family [ConnectX-5] |
| 25G Switch | Mellanox Onyx MSN2700 |
Test Tools
| Tools | Purposes |
| netperf 2.7 | For latency test. |
| iperf3 3.1.7 | For I/O throughput test. |
| Iperf 2.0.7 |
PCI Pass-through Test
Hardware Configuration
Device 1: Physical Server for Solarflare NIC Test
| Hardware | Configuration |
| CPU (Server A) | Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz 40 CPUs |
| NIC (Server A) | Mellanox ConnectX-4 Lx |
| CPU (Server B) | Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz 64 CPUs |
| NIC (Server B) | Mellanox ConnectX-4 Lx |
| Switch (25G NIC) | Mellanox SN2010 |
Device 2: Physical Server for Mellanox NIC Test
| Hardware | Configuration |
| CPU (Server A) | Intel(R) Xeon(R) Gold 6238 CPU @ 2.10GHz 88 CPUs |
| NIC (Server A) | Solarflare XtremeScale X2522-25G Adapter * 2 |
| CPU (Server B) | Intel(R) Xeon(R) Gold 6238 CPU @ 2.10GHz 88 CPUs |
| NIC (Server B) | Solarflare XtremeScale X2522-25G Adapter * 2 |
| Switch (10G NIC) | Mellanox SX1410 |
| Switch (25G NIC) | Mellanox SX2010 |
Test Tool
| Tool | Purpose |
| sfnettest sr-916760.1 | For latency test. |
