kvsoftware

Sep – Dec 2025 · coursework · Guriang HPC

Performance Analysis MPI on K8s

Does MPI pay a price for running on Kubernetes? Benchmarked against bare metal

A study of what containerising HPC workloads costs. The same MPI jobs, a LAMMPS molecular-dynamics simulation (500,000 atoms) and the OSU latency benchmark, ran at 1 to 32 processes on bare metal and on Kubernetes, and an interactive dashboard compares runtime, scaling, and message latency side by side.

Data Analyst, team of 3 · MPI · Kubernetes · LAMMPS · Python · Dash · Plotly

High-performance computing has always run on bare metal, as close to the processor as possible. But Kubernetes is spreading into HPC for its portability and orchestration, so the question is no longer "can we run MPI on Kubernetes?" but "what does that flexibility cost?" We measured it.

Throughput at 32 cores
+8.8%
Network overhead
−2.4%
Cost
−6.9%

The setup

Same machine for both, to keep it fair: earth04, a node of Universitas Padjadjaran's Guriang HPC facility, with 64 cores and 128 GB of RAM.

Bare metal

OpenMPI 4.0.3 installed straight on the OS, talking over TCP/IP with no virtualization layer.

Kubernetes

Minikube 1.37 with the Docker driver, running the jobs through the Kubeflow MPI Operator.

  1. LAMMPS

    A Lennard-Jones melt: 500,000 atoms, 1,000 timesteps.

  2. 1 to 32 cores

    Run on bare metal and on Kubernetes.

  3. OSU benchmarks

    Latency and bandwidth between processes.

  4. Dashboard

    Throughput, overhead, and resources side by side.

The same workloads, run in both environments and compared.

What we found

We expected Kubernetes to cost performance. At the top of the range, it didn't: at 32 cores Kubernetes ran 45.7 timesteps per second, 8.8% ahead of bare metal, with less network overhead and less of the CPU spent on communication. Averaged across 2 to 32 cores, the picture is closer, with an average slowdown of 7.6%.

The HPC benchmark dashboard: summary cards for slowdown, cost, network overhead, and throughput at 32 cores, a performance-scaling chart, and an OSU latency chart comparing Kubernetes and bare metal
The dashboard: the headline numbers, throughput as cores scale, and latency between processes.

The OSU benchmark showed bandwidth peaking at 32 KB and 64 KB messages, the same shape reported in earlier research on HPC over Kubernetes (Beltre et al., 2019).

Why it might be faster

Our best guess, and only a guess: Kubernetes puts every worker in its own Linux control group, which may let each one use the L1 and L2 caches more effectively. On bare metal, the OS scheduler treats MPI ranks like any other process.

The catch

This ran on a single node. Every message between pods crossed a virtual bridge in the kernel, never a physical network. Real Kubernetes clusters span many nodes, where traffic picks up physical latency and the overhead of the container network. The result needs testing on a multi-node cluster before anyone migrates on the strength of it.

My part

On a team of three, I owned the analysis and the dashboard: turning the benchmark runs into the comparison above, in Python with Dash and Plotly.

Back to the work