Performance testing with Compute Engine clients

This page describes how to test the performance of a Google Cloud Managed Lustre instance using Compute Engine clients. It provides instructions for measuring single-client performance using fio and aggregate multi-client performance using the IOR benchmark tool.

Measuring single-client performance

To test read and write performance from a single Compute Engine client, use the fio (Flexible I/O tester) command line tool.

1. Install fio:

   Rocky 8

   sudo dnf install fio -y
   

   Ubuntu 20.04 and 22.04

   sudo apt update
   sudo install fio
   

2. Run the following command:

   fio --ioengine=libaio --filesize=32G --ramp_time=2s \
   --runtime=5m --numjobs=16 --direct=1 --verify=0 --randrepeat=0 \
   --group_reporting --directory=/lustre --buffer_compress_percentage=50 \
   --name=read --blocksize=1m --iodepth=64 --readwrite=read
   

The test takes approximately 5 minutes to complete. When finished, the results are displayed. Depending on your configuration, you can expect throughput up to your VM's maximum network speed, and thousands of IOPS per TiB.

Measuring multi-client performance

To test Managed Lustre's read and write performance from multiple Compute Engine clients, use the IOR benchmark tool. The following instructions show you how to automate your client setup and use IOR to test aggregate I/O from multiple client machines. IOR uses MPI - a message-passing protocol - to enable the multiple client machines to coordinate with each other.

Before you begin, ensure that your network's mtu value is set to 8896.

Set environment variables and generate an SSH key

Before deploying your cluster, generate an SSH key on your local machine. This key will be distributed to your client machines during creation to enable passwordless communication for MPI.

export SSH_USER="lustre-user"
export CLIENT_PREFIX="lustre-client"

# Generate an SSH key for the specified user
ssh-keygen -t rsa -b 4096 -C "${SSH_USER}" -N '' -f "./id_rsa"
chmod 600 "./id_rsa"

# Create a metadata file formatted for Google Cloud
echo "${SSH_USER}:$(cat "./id_rsa.pub") ${SSH_USER}" > "./keys.txt"

Create the startup script

Save the following content into a file named install-ior.sh on your local machine. This script detects the operating system, waits for boot locks to release, safely installs the Lustre client and dependencies, compiles a stable release of IOR, and mounts the Managed Lustre file system.

Replace LUSTRE_IP with your Managed Lustre instance's IP address, and FS_NAME with your file system name.

#!/bin/bash
source /etc/os-release

if [[ "$ID" == "ubuntu" ]]; then
    # Ubuntu
    # Wait for apt lock
    while fuser /var/lib/dpkg/lock-frontend >/dev/null 2>&1; do sleep 5; done

    # Configure Artifact Registry repo for Ubuntu
    curl -fsSL https://us-apt.pkg.dev/doc/repo-signing-key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/us-apt-pkg-dev.gpg

    if [[ "$VERSION_ID" == "22.04" ]]; then
        REPO_NAME="lustre-client-ubuntu-jammy"
    elif [[ "$VERSION_ID" == "24.04" ]]; then
        REPO_NAME="lustre-client-ubuntu-noble"
    fi

    echo "deb [signed-by=/usr/share/keyrings/us-apt-pkg-dev.gpg] https://us-apt.pkg.dev/projects/lustre-client-binaries $REPO_NAME main" | sudo tee /etc/apt/sources.list.d/lustre-client.list

    sudo apt-get update
    while ! sudo apt-get install -y lustre-client-modules-$(uname -r) lustre-client-utils openmpi-bin libopenmpi-dev make gcc g++ wget git automake autoconf libaio-dev; do
        sleep 5
    done
    sudo modprobe lustre
else
    # Red Hat / Rocky Linux
    while systemctl is-active --quiet dnf-makecache.service; do sleep 5; done

    if [[ "$ID" == "rocky" && "$VERSION_ID" == 8* ]]; then
        REPO="lustre-client-rocky-8"
    elif [[ "$ID" == "rocky" && "$VERSION_ID" == 9* ]]; then
        REPO="lustre-client-rocky-9"
    elif