Kubernetes on VMs vs. Bare Metal: Comparison of Performance

In our previous blog post, “Kubernetes on VMs vs. Bare Metal: Comparison of Architecture and Capabilities”, we explored the distinctions between deploying Kubernetes on VMs and bare metals, focusing on IT architecture and features. However, there is a lack of exact data references regarding how Kubernetes performs in these two environments under different use cases.

In this article, we aim to address this gap by presenting empirical performance data of Kubernetes running stateful and stateless applications on VMs (SMTX Kubernetes Services, referred to as SKS) and bare-metal servers. Overall, the results demonstrate that SKS shows performance ranging from 82% to 96% of bare-metal-based Kubernetes, indicating that VM-based Kubernetes can effectively meet the requirements of the majority of containerized applications in production environments.

1 Test Objectives

Under the same Kubernetes version, tuning parameters, and application resource configurations, we tested how Kubernetes performs on SKS (deployed on SmartX HCI) and bare metals in the lab environment as well as running stateful and stateless applications. We selected MySQL, Redis, and Kafka for stateful application tests, and Nginx and Online Boutique for stateless application tests.

We utilized bare-metal servers with identical configurations (including CPU, memory, local disk, and network) to support two Kubernetes clusters: SKS cluster and bare-metal Kubernetes cluster. SKS cluster, consisting of one Control Plane and one Worker, was deployed on SmartX HCI (with native hypervisor ELF). While the SKS cluster utilized the distributed storage of SmartX HCI, the bare-metal Kubernetes cluster used a separate distributed storage cluster through CSI. The configuration of this distributed storage cluster (CPU/memory/local disk/network) remained consistent with that of SKS cluster.

For ease of expression, the two testing environments mentioned below will be referred to as “SKS” and “bare-metal Kubernetes” respectively.

2 Test Environment Configurations

2.1 Software Configurations

2.1.1 SKS Environment (Deployed on SmartX HCI)

SMTX OS (SmartX HCI Software)：

SKS：

2.1.2 Bare-Metal Environment

2.2 Hardware Configurations

Bare-metal servers：

3 Test Results

3.1 Benchmark Test

We conducted FIO stress tests in both environments. The results showed that SKS outperformed bare-metal Kubernetes, probably due to the I/O localization feature of SmartX HCI.

3.2 Stateful Application Tests

3.2.1 MySQL

Software Configuration

Test Results

ReadWrite:

Continuous 1-hour testing:

3.2.2 Redis

Parameters

10,000,000 requests
200 parallel clients
Mixed mode data persistence (AOF, RDB)

Test Results

3.2.3 Kafka

Parameters

6 Partition 3 Replication
num-records 10,000,000

Test Results

3.3 Stateless Application Tests

3.3.1 Nginx

Software Configuration

Test Results

With less than 20,000 concurrent users, the performance gap between SKS and bare-metal Kubernetes was less than 1%.

After continuous running, with 25,000 concurrent users, the performance gap between SKS and bare-metal Kubernetes was around 12%.

3.3.2 Online Boutique

Online Boutique is a cloud-native application for microservice demonstration, consisting of 11 layers of microservice applications. It simulates a web-based e-commerce application where users can browse products, add them to the shopping cart, and make purchases.

Software Configurations

Test Results

With less than 5,000 concurrent users, the performance gap between SKS and bare-metal Kubernetes was less than 4%. As the number of users increased, this gap could reach around 20%.

4 Conclusion

In general, SKS demonstrates a remarkable performance level, achieving between 82% and 96% of the performance observed in bare-metal Kubernetes deployments. Particularly in common business scenarios, SKS and bare-metal Kubernetes exhibit nearly identical performance, highlighting SKS’s ability to meet the performance requirements of the majority of containerized applications. Specifically:

SKS can well support containerized applications for enterprises in the early and initial stages.
SKS can effectively support containerized applications on Dev/Test clusters in all scenarios.
SKS can meet the performance requirements of most common application use cases unless the application has stringent performance requirements or a large number of concurrent users. In the next version, SKS will upgrade its capability to support bare metal deployment.
It’s worth noting that in the tests above, SKS was deployed on SmartX HCI and enabled Boost mode for I/O acceleration. It does not guarantee that other virtualization/HCI platforms can achieve the same results.

Additionally, although there is a performance gap between VM-based and bare-metal-based Kubernetes, Gartner* expects that 70% of data centers x86 workloads will still be deployed in virtualized environments through 2027 due to virtualization’s benefits in resource utilization, elastic scaling, security isolation, and simplicity of operations and maintenance.