AI Hypercomputer provides vertically integrated infrastructure optimized for demanding artificial intelligence (AI) and machine learning (ML) workloads. High-performance NVIDIA GPUs are categorized by how the system handles their lifecycle events and maintenance: General GPUs and Clustered GPUs.
To identify the optimal infrastructure category for your workload, see Choose your accelerator infrastructure.
General GPUs
General GPUs are managed as independent compute units, by using standard Compute Engine and Google Kubernetes Engine (GKE) APIs to integrate directly with Google Cloud's control plane.
General GPUs include the following machine series:
- A3 High with 1, 2, or 4 (NVIDIA H100) GPUs: optimized for standard training and inference workloads that don't require an 8-GPU synchronized cluster.
- A3 Edge (NVIDIA H100): optimized for inference that is distributed across multiple hosts with high throughput using standard VPC networking.
- A2 (NVIDIA A100): optimized for high-performance single-node serving and small-scale fine-tuning.
- G4 (NVIDIA RTX PRO 6000): optimized for cost-effective entry-level inference and graphics workloads.
- G2 (NVIDIA L4): optimized for mainstream inference and RAG.
- N1 (NVIDIA T4 or V100): optimized for entry-level inference and research tasks that prioritize cost over high-performance interconnects.
Clustered GPUs
Clustered GPUs are managed as a single, tightly coupled system of thousands of interconnected accelerators using Cluster Director.
Clustered GPUs include the following machine series:
- A4X Max (NVIDIA GB300): designed for bare-metal exascale training and high performance computing, featuring NVIDIA GB300 Grace Blackwell Ultra Superchips.
- A4X (NVIDIA GB200) and A4 (NVIDIA B200): designed for foundation model training and exascale foundation model training.
- A3 Ultra (NVIDIA H200), A3 Mega (NVIDIA H100), and A3 High with 8 (NVIDIA H100) GPUs: optimized for large-scale distributed training and multi-host inference involving massive data tables.
GPU performance considerations
To evaluate and optimize your workload's performance, you can track hardware-level metrics such as CUDA core and Tensor Core utilization. To monitor your resources and evaluate performance across both General and Clustered environments, see the following:
- Monitoring dashboards: track real-time GPU utilization and memory usage for individual standalone instances or orchestrated cluster fleets.
- Goodput metrics recipes: measure the percentage of time your accelerators perform useful computation versus system overhead, which is highly recommended for large-scale distributed training on clustered hardware.
What's next
- To evaluate your infrastructure requirements, see Choose your accelerator infrastructure.
- To map workloads to accelerators, see Identify your workload needs.