This document describes best practices for creating a secure and resilient networking environment for AI Hypercomputer workloads. These recommendations are intended for network architects, network engineers, and developers who want to configure and deploy artificial intelligence (AI) and machine-learning (ML) workloads on AI Hypercomputer.
Establish clear and restricted IAM roles
Configuring IAM correctly helps to improve the security and
success of your AI Hypercomputer deployments. In production
environments, inadequate or misconfigured permissions can lead to deployment
failures. AI Hypercomputer deployments, especially those using
Cluster Toolkit, often fail in
environments with hardened security postures where the default Compute Engine
service account does not have the broad Editor role.
To help mitigate deployment issues that might occur due to permission issues, follow best practices listed in this section.
Use dedicated service accounts
For better security and control, avoid using the default Compute Engine service account. Instead, create a dedicated service account for your AI Hypercomputer deployment.
You can use Managed Workload Identity to authenticate and authorize workloads instead of using service account tokens. For more information, see Authenticate workloads using mTLS for Compute Engine or Workload Identity for GKE.
Grant necessary IAM roles
Grant the following IAM roles to the dedicated service account you created:
- Compute Admin (
roles/compute.admin): provides full control of Compute Engine resources. - Service Account User (
roles/iam.serviceAccountUser): allows the service account to be attached to other resources, which is crucial for tools like Packer when building custom images. - Storage Admin (
roles/storage.admin): requires access to and management of Cloud Storage buckets, for example, to store Packer images or other artifacts. - Logging Admin (
roles/logging.admin): allows the service account to configure logging and view logs, which is essential for debugging.
Verify permissions before deployment
Before you start a deployment, verify that your service account has the
necessary permissions. Run the gcloud projects get-iam-policy
command:
gcloud projects get-iam-policy PROJECT_ID \
--flatten="bindings[].members"