This document summarizes how you create a cluster for your AI workloads on AI Hypercomputer. Specifically, this document guides you through the process and choices to make when you start a cluster. As an alternative to manual evaluation, you can prompt Gemini in the Google Cloud console to compare consumption options for your workload and budget. For more information, see Design your AI infrastructure with Gemini.
This document assumes that you are familiar with commonly used terminology for AI and ML workloads, such as model training and inference. It also assumes that you have identified the specific AI workload for which you need to determine the optimal machine type and capacity needs for your deployment—for example, foundation model pre-training, fine-tuning, or inference.
Start a cluster
Starting a cluster involves the following steps:
- Determine your workload and choose a machine type
- Choose a consumption option and obtain capacity
- Choose a deployment option
- Choose an orchestrator
- Choose the operating system and cluster image
- Create your cluster
- Provision storage for your workload
Determine your workload and choose a machine type
Select a machine type for your AI workload. AI Hypercomputer supports cluster creation for both clustered GPUs and general GPUs.
To help you choose, determine your workload requirements and match them with the recommended machine type and GPU type:
- Clustered GPUs: best for large-scale, high-performance workloads such as pre-training foundation models, fine-tuning large models, and inference across multiple hosts.
- General GPUs: best for mainstream inference and serving, retrieval-augmented generation (RAG), and cost-effective small-to-medium model training and fine-tuning.
To match your workload to the recommended machine type, use this table:
| GPU type | Workload or use case | Recommended machine types |
|---|---|---|
| Clustered GPU | Pre-training foundation models and inference across multiple hosts | A4X Max (NVIDIA GB300)*, A4X (NVIDIA GB200)* |
| Large model training, fine-tuning, and inference | A4 (NVIDIA B200), A3 Ultra (NVIDIA H200 141GB) | |
| Mainstream model inference and fine-tuning | A3 Mega (NVIDIA H100 80GB), A3 High (NVIDIA H100 80GB) | |
| General GPU | High-throughput edge serving and inference | A3 Edge (NVIDIA H100 80GB) |
| High-performance single-node serving and small-scale fine-tuning | A2 (NVIDIA A100) | |
| Cost-optimized entry-level inference | G4 (NVIDIA RTX PRO 6000), N1 (NVIDIA T4 or V100) | |
| Mainstream inference, RAG, and small-to-medium model training | G2 (NVIDIA L4) |
For detailed information about each machine series, see About GPU accelerators.
Choose a consumption option and obtain capacity
Select a consumption option for your GPU resources based on your chosen machine type and whether you use General GPUs or Clustered GPUs.
Consumption options for general GPUs
| Consumption option | Available for | Best for | How to request |
|---|---|---|---|
| On-demand | All general GPUs. | Workloads that don't require assured capacity. | Create a compute instance or cluster and specify the standard provisioning model. For instructions, see Create VM instances. Tip: To increase your chances of obtaining general GPU capacity, use Flex-start or Spot. |
| Standard reservations and standard future reservations | All general GPUs. |