Performance testing on Google Kubernetes Engine

To test a Google Kubernetes Engine (GKE) workload's read and write performance from multiple GKE clients, use the IOR benchmark tool. The following instructions show you how to automate your client setup and use IOR with mpirun over passwordless SSH between Kubernetes Pods to test aggregate I/O.

Prerequisites

  • A Managed Lustre instance already provisioned.

  • A local Docker environment configured and authenticated to push to Google Artifact Registry or Container Registry (see Authentication methods).

  • Ensure that your network's mtu value is set to 8896.

Create a GKE cluster

To test performance, you need a GKE cluster with the Managed Lustre CSI driver enabled. For high-performance storage workloads, configure your GKE node pools with compute-optimized machine families (e.g., c2 or c3) and TIER_1 networking.

Run the following command to create a Standard GKE cluster optimized for performance testing:

gcloud container clusters create CLUSTER_NAME \
    --zone=ZONE \
    --machine-type=MACHINE_TYPE \
    --addons=LustreCsiDriver \
    --network-performance-configs=total-egress-bandwidth-tier=TIER_1 \
    --network=NETWORK \
    --num-nodes=NUM_NODES

  • Replace ZONE and NETWORK with your specific deployment values. The cluster must reside in the same VPC network as your Managed Lustre instance.

  • Choose a MACHINE_TYPE. See Performance considerations for information on choosing machine types to obtain the best throughput.

  • If your machine type doesn't support TIER_1 networking, delete the --network-performance-configs line from the command.

  • Specify the NUM_NODES. To saturate your file system, your cluster's aggregate network capacity should exceed your file system's provisioned throughput by ~20%.

    For machines with Tier 1 networking enabled, a single node can push between 25 Gbps–200 Gbps (~3,000–25,000 MBps), depending on the VM family and the CPU count. For standard instances, egress is typically capped around 2 Gbps per vCPU.

    For example, if your Managed Lustre instance capacity yields 100,000 MBps of theoretical throughput, you need an aggregate client egress of 120,000 MBps (100,000 * 1.2) to saturate it:

    • With standard instances: If each node has a published egress of 2,000 MBps, you should provision at least 60 nodes (120,000 / 2,000).
    • With Tier 1 networking: If each node has a published egress of 10,000 MBps (~80 Gbps), you should provision at least 12 nodes (120,000 / 10,000).

Create the IOR Docker Image

Build a container image with OpenMPI and IOR installed. Compile IOR with Asynchronous I/O (AIO) support for better performance.

  1. Create a file named Dockerfile locally:

    FROM ubuntu:22.04

# Prevent interactive prompts during installation
ENV DEBIAN_FRONTEND=noninteractive

# Install dependencies, SSH, and required Autotools packages
RUN apt-get update && apt-get install -y \
  openssh-server