Performance testing with Compute Engine clients

This page describes how to test the performance of a Google Cloud Managed Lustre instance using Compute Engine clients. It provides instructions for measuring single-client performance using fio and aggregate multi-client performance using the IOR benchmark tool.

Measuring single-client performance

To test read and write performance from a single Compute Engine client, use the fio (Flexible I/O tester) command line tool.

Install fio:

Rocky 8

sudo dnf install fio -y

Ubuntu 20.04 and 22.04

sudo apt update
sudo install fio

Run the following command:

fio --ioengine=libaio --filesize=32G --ramp_time=2s \
--runtime=5m --numjobs=16 --direct=1 --verify=0 --randrepeat=0 \
--group_reporting --directory=/lustre --buffer_compress_percentage=50 \
--name=read --blocksize=1m --iodepth=64 --readwrite=read

The test takes approximately 5 minutes to complete. When finished, the results are displayed. Depending on your configuration, you can expect throughput up to your VM's maximum network speed, and thousands of IOPS per TiB.

Measuring multi-client performance

To test Managed Lustre's read and write performance from multiple Compute Engine clients, use the IOR benchmark tool. The following instructions show you how to automate your client setup and use IOR to test aggregate I/O from multiple client machines. IOR uses MPI - a message-passing protocol - to enable the multiple client machines to coordinate with each other.

Before you begin, ensure that your network's mtu value is set to 8896.

Set environment variables and generate an SSH key

Before deploying your cluster, generate an SSH key on your local machine. This key will be distributed to your client machines during creation to enable passwordless communication for MPI.

export SSH_USER="lustre-user"
export CLIENT_PREFIX="lustre-client"

# Generate an SSH key for the specified user
ssh-keygen -t rsa -b 4096 -C "${SSH_USER}" -N '' -f "./id_rsa"
chmod 600 "./id_rsa"

# Create a metadata file formatted for Google Cloud
echo "${SSH_USER}:$(cat "./id_rsa.pub") ${SSH_USER}" > "./keys.txt"

Create the startup script

Save the following content into a file named install-ior.sh on your local machine. This script detects the operating system, waits for boot locks to release, safely installs the Lustre client and dependencies, compiles a stable release of IOR, and mounts the Managed Lustre file system.

Replace LUSTRE_IP with your Managed Lustre instance's IP address, and FS_NAME with your file system name.

#!/bin/bash
source /etc/os-release

if [[ "$ID" == "ubuntu" ]]; then
    # Ubuntu
    # Wait for apt lock
    while fuser /var/lib/dpkg/lock-frontend >/dev/null 2>&1; do sleep 5; done

    # Configure Artifact Registry repo for Ubuntu
    curl -fsSL https://us-apt.pkg.dev/doc/repo-signing-key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/us-apt-pkg-dev.gpg

    if [[ "$VERSION_ID" == "22.04" ]]; then
        REPO_NAME="lustre-client-ubuntu-jammy"
    elif [[ "$VERSION_ID" == "24.04" ]]; then
        REPO_NAME="lustre-client-ubuntu-noble"
    fi

    echo "deb [signed-by=/usr/share/keyrings/us-apt-pkg-dev.gpg] https://us-apt.pkg.dev/projects/lustre-client-binaries $REPO_NAME main" | sudo tee /etc/apt/sources.list.d/lustre-client.list

    sudo apt-get update
    while ! sudo apt-get install -y lustre-client-modules-$(uname -r) lustre-client-utils openmpi-bin libopenmpi-dev make gcc g++ wget git automake autoconf libaio-dev; do
        sleep 5
    done
    sudo modprobe lustre
else
    # Red Hat / Rocky Linux
    while systemctl is-active --quiet dnf-makecache.service; do sleep 5; done

    if [[ "$ID" == "rocky" && "$VERSION_ID" == 8* ]]; then
        REPO="lustre-client-rocky-8"
    elif [[ "$ID" == "rocky" && "$VERSION_ID" == 9* ]]; then
        REPO="lustre-client-rocky-9"
    elif [[ "$ID" == "rhel" && "$VERSION_ID" == 9* ]]; then
        REPO="lustre-client-rocky-9"
    fi

    gcloud beta artifacts print-settings yum --repository=$REPO --location=us --project=lustre-client-binaries | sudo bash

    while ! sudo dnf -y --enablerepo=$REPO install kmod-lustre-client lustre-client; do sleep 5; done
    sudo modprobe lustre

    while ! sudo dnf install -y openmpi openmpi-devel make gcc gcc-c++ wget git automake autoconf libaio-devel; do sleep 5; done

    export PATH=$PATH:/usr/lib64/openmpi/bin
fi

# Build IOR from source
echo "Cloning repo: https://github.com/hpc/ior.git and building IOR"
pushd /tmp
git clone -b 4.0.0 https://github.com/hpc/ior
cd ior
./bootstrap
./configure --disable-dependency-tracking --with-aio
make clean
make -j"$(nproc)"
sudo make install
popd
echo "Finished building IOR"

# Mount the Managed Lustre file system
mkdir -p /lustre

if ! grep -q "/lustre" /etc/fstab; then
    echo "LUSTRE_IP@tcp:/FS_NAME /lustre lustre defaults,_netdev 0 0" >> /etc/fstab
fi

mount -a

Deploy the client machines

Run the following command to bulk create the Compute Engine client machines.

gcloud compute instances bulk create \
  --name-pattern="${CLIENT_PREFIX}-####" \
  --zone="ZONE" \
  --machine-type="MACHINE_TYPE" \
  --scopes="https://www.googleapis.com/auth/cloud-platform" \
  --network-interface=subnet=SUBNET,nic-type=GVNIC \
  --network-performance-configs=total-egress-bandwidth-tier=TIER_1 \
  --metadata-from-file=ssh-keys=./keys.txt,startup-script=install-ior.sh \
  --create-disk=auto-delete=yes,boot=yes,\
image-family=IMAGE_FAMILY,\
image-project=IMAGE_PROJECT,\
mode=rw,size=100,type=DISK_TYPE \
  --count NUM_NODES

Replace ZONE and SUBNET with your specific deployment values.
Choose a MACHINE_TYPE. The overall performance of your benchmark depends on the client machine types. See Performance considerations for information on choosing machine types to obtain the best throughput.
If your machine type doesn't support TIER_1 networking, delete the --network-performance-configs line from the command.
Set the DISK_TYPE to one of hyperdisk-balanced (for a machine type with a 4 in its generation name, such as c4a or n4) or pd-balanced.

Specify the IMAGE_FAMILY and IMAGE_PROJECT. Supported values are:

OS	Image family (x86)	Image family (ARM)	Image project
HPC Rocky Linux 8	`hpc-rocky-linux-8`	Not supported	`cloud-hpc-image-public`
Rocky Linux 9	`rocky-linux-9`	`rocky-linux-9-arm64`	`rocky-linux-cloud`
RHEL 9	`rhel-9`	`rhel-9-arm64`	`rhel-cloud`
Ubuntu 22.04 LTS	`ubuntu-2204-lts`	`ubuntu-2204-lts-arm64`	`ubuntu-os-cloud`
Ubuntu 24.04 LTS	Not supported	`ubuntu-2404-lts-arm64`	`ubuntu-os-cloud`

Specify the NUM_NODES. To saturate your file system, your cluster's aggregate network capacity should exceed your file system's provisioned throughput by ~20%.

For machines with Tier 1 networking enabled, a single machine can push between 25 Gbps–200 Gbps (~3,000–25,000 MBps), depending on the VM family and the CPU count. For standard instances, egress is typically capped around 2 Gbps per vCPU.

For example, if your Managed Lustre instance capacity yields 100,000 MBps of theoretical throughput, you need an aggregate client egress of 120,000 MBps (100,000 * 1.2) to saturate it:
- With standard instances: If each client machine has a published egress of 2,000 MBps, you should provision at least 60 clients (120,000 / 2,000).
- With Tier 1 networking: If each client machine has a published egress of 10,000 MBps (~80 Gbps), you should provision at least 12 clients (120,000 / 10,000).

Copy keys and files

While you wait for the client machines to finish their startup scripts, run the following commands locally to automatically configure all nodes for inter-node communication.

Save the client machines' private IP addresses for the MPI host file, and the public IP addresses for SSH and SCP access to the client machines:

gcloud compute instances list --filter="name ~ '^${CLIENT_PREFIX}*'" --format="csv[no-heading](INTERNAL_IP)" > hosts.txt
gcloud compute instances list --filter="name ~ '^${CLIENT_PREFIX}*'" --format="csv[no-heading](EXTERNAL_IP)" > external_ips.txt

Copy the private key to all clients. This allows the worker nodes to communicate with each other securely during the benchmark:

while IFS= read -r IP || [[ -n "$IP" ]]
do
  [[ -z "$IP" ]] && continue
  echo "Preparing and copying to ${SSH_USER}@${IP}..."
  # Ensure the .ssh directory exists
  ssh -i ./id_rsa -o StrictHostKeyChecking=no "${SSH_USER}@${IP}" "mkdir -p ~/.ssh && chmod 700 ~/.ssh"
  # Copy the file
  scp -i ./id_rsa -o StrictHostKeyChecking=no ./id_rsa "${SSH_USER}@${IP}:~/.ssh/id_rsa"
done < "./external_ips.txt"

Designate the head node and copy the host file to it. This will be the machine from which you run the benchmark.

export HEAD_NODE=$(head -n 1 ./external_ips.txt)
scp -i ./id_rsa -o "StrictHostKeyChecking=no" -o UserKnownHostsFile=/dev/null ./hosts.txt ${SSH_USER}@${HEAD_NODE}:~/hostfile

Connect and verify

Connect to your head node:

ssh -i ./id_rsa -o "StrictHostKeyChecking=no" -o UserKnownHostsFile=/dev/null ${SSH_USER}@${HEAD_NODE}

Verify that the startup script has finished and the file system is successfully mounted before running the benchmark.

To check the startup script logs:
```
sudo journalctl -u google-startup-scripts.service -f
```
To verify the file system is mounted:
```
df -h | grep lustre
```
If the file system is not listed, wait a few more minutes for the background installation script to complete. Once mounted, it will be available at the absolute path /lustre.

Run the IOR benchmark

Rocky Linux and RHEL only: Load the OpenMPI module from the head node.

if [ -f /etc/profile.d/modules.sh ]; then
    source /etc/profile.d/modules.sh
    module load mpi/openmpi-$(arch)
fi

Create a test directory and take ownership:

sudo mkdir -p /lustre/test
sudo chown -R $USER:$USER /lustre/test

Define test variables:
```
export NUM_NODES="NUM_NODES"
export PROCESSES_PER_NODE="PROCESSES_PER_NODE"
export NUM_PROCESSES=$(( NUM_NODES * PROCESSES_PER_NODE ))
```
Where:
- NUM_NODES: The total number of client machines participating in the test.
- PROCESSES_PER_NODE: The number of MPI ranks to run on each individual client machine. We recommend that you start by setting this to match the number of physical cores (or half the number of vCPUs) on your client machines. For high-performance machine types, setting this between 8 and 16 typically yields the best network throughput. Setting this too high can cause context-switching overhead and degrade benchmark performance.

Run the command to start the benchmark:

Rocky Linux and RHEL

Write throughput

This command writes continuously for 60 seconds to test peak steady-state throughput. The per-task file size is set to an arbitrarily large 50 TiB ceiling to keep tasks writing until the 60-second timer expires.

mpirun \
  --allow-run-as-root \
  --mca plm_rsh_no_tree_spawn 1 \
  --mca opal_set_max_sys_limits 1 \
  --mca plm_rsh_num_concurrent ${NUM_NODES} \
  --mca plm_rsh_args "-o StrictHostKeyChecking=no" \
  --prefix /usr/lib64/openmpi \
  --npernode ${PROCESSES_PER_NODE} \
  --np ${NUM_PROCESSES} \
  --hostfile ~/hostfile \
  /usr/local/bin/ior \
  -a AIO \
  --posix.odirect \
  -F -g -v -w -k \
  -t 4m -b 50t \
  -D 60 \
  -O stoneWallingWearOut=1 \
  -O stoneWallingStatusFile=/lustre/test/ior-easy.stonewall \
  -o /lustre/test/ior_file

Read throughput

This phase reads back exactly the amount of data that was successfully written during the 60-second write throughput test. Although the per-task file size (-b) is set to 50 TiB to match the write phase's geometry, the -O stoneWallingWearOut=1 flag instructs IOR to stop reading as soon as it hits the exact data boundary recorded in the stonewall status file.

mpirun \
  --allow-run-as-root \
  --mca plm_rsh_no_tree_spawn 1 \
  --mca opal_set_max_sys_limits 1 \
  --mca plm_rsh_num_concurrent ${NUM_NODES} \
  --mca plm_rsh_args "-o StrictHostKeyChecking=no" \
  --prefix /usr/lib64/openmpi \
  --npernode ${PROCESSES_PER_NODE} \
  --np ${NUM_PROCESSES} \
  --hostfile ~/hostfile \
  /usr/local/bin/ior \
  -a AIO \
  --posix.odirect \
  -F -g -v -r -k \
  -t 4m -b 50t \
  -D 60 \
  -O stoneWallingWearOut=1 \
  -O stoneWallingStatusFile=/lustre/test/ior-easy.stonewall \
  -o /lustre/test/ior_file

Write IOPS

This test uses small 4 KiB transfer sizes and 8 GiB per-task file sizes to measure the maximum Input/Output Operations Per Second (IOPS) the file system can handle.

mpirun \
  --allow-run-as-root \
  --mca plm_rsh_no_tree_spawn 1 \
  --mca opal_set_max_sys_limits 1 \
  --mca plm_rsh_num_concurrent ${NUM_NODES} \
  --mca plm_rsh_args "-o StrictHostKeyChecking=no" \
  --prefix /usr/lib64/openmpi \
  --np ${NUM_PROCESSES} \
  --oversubscribe \
  --map-by node \
  --bind-to socket \
  --hostfile ~/hostfile \
  /usr/local/bin/ior \
  -e \
  -t 4k \
  -b 8g \
  -s 1 \
  -a AIO \
  --posix.odirect \
  --aio.max-pending=256 \
  -w \
  -F \
  -z \
  -Q 1 \
  -G 1745405099 \
  -D 45 \
  -O stoneWallingWearOut=1 \
  -o /lustre/test/ior-random

Read IOPS

To prevent reading from a sparse file, this test uses two commands: a write to create a solid file using 4 MiB transfer sizes and 8 GiB per-task file sizes, followed by the actual 4 KiB random read IOPS test.

Create the file to read:

mpirun \
  --allow-run-as-root \
  --mca plm_rsh_no_tree_spawn 1 \
  --mca opal_set_max_sys_limits 1 \
  --mca plm_rsh_num_concurrent ${NUM_NODES} \
  --mca plm_rsh_args "-o StrictHostKeyChecking=no" \
  --prefix /usr/lib64/openmpi \
  --npernode ${PROCESSES_PER_NODE} \
  --np ${NUM_PROCESSES} \
  --oversubscribe \
  --map-by node \
  --bind-to socket \
  --hostfile ~/hostfile \
  /usr/local/bin/ior \
  -a AIO \
  --posix.odirect \
  -w \
  -F \
  -k \
  -t 4m \
  -b 8g \
  -s 1 \
  -Q 1 \
  -G 1745405099 \
  -o /lustre/test/ior_rand_read

Run the IOR read test:

mpirun \
  --allow-run-as-root \
  --mca plm_rsh_no_tree_spawn 1 \
  --mca opal_set_max_sys_limits 1 \
  --mca plm_rsh_num_concurrent ${NUM_NODES} \
  --mca plm_rsh_args "-o StrictHostKeyChecking=no" \
  --prefix /usr/lib64/openmpi \
  --oversubscribe \
  --map-by node \
  --bind-to socket \
  --npernode ${PROCESSES_PER_NODE} \
  --np ${NUM_PROCESSES} \
  --hostfile ~/hostfile \
  /usr/local/bin/ior \
  -a AIO \
  --posix.odirect \
  --aio.max-pending 256 \
  -r \
  -F \
  -z \
  -t 4k \
  -b 8g \
  -s 1 \
  -Q 1 \
  -G 1745405099 \
  -D 45 \
  -O stoneWallingWearOut=1 \
  -o /lustre/test/ior_rand_read

Ubuntu