Get started with Batch

This page describes how to get started with Batch for Google Cloud.

Overview

Batch is a fully managed service that lets you schedule, queue, and execute batch processing workloads on Google Cloud resources. For example, consider using Batch for high performance computing (HPC), machine learning (ML), and data-processing workloads. Batch provisions resources and manages capacity on your behalf, allowing your batch workloads to run at scale.

Using Batch, you don't need to configure and manage third-party job schedulers, provision and deprovision resources, or request resources one zone at a time. To run a job, you specify parameters for the resources required for your workload, then Batch obtains resources and queues the job for execution. Batch provides native integration with other Google Cloud services to aid in the scheduling, execution, storage, and analysis of batch jobs, so you can focus on submitting a job and consuming the results.

Batch consists of the following components:

  • Job: a scheduled program that runs a set of tasks to completion without any user interaction, typically for computational workloads. For example, a job might be a single shell script or a complex, multipart computation.

    Specifically, a Batch job represents an array of one or more tasks and the environment to run those tasks in. You define the program for the job as a sequence of one or more runnables. Each task runs the sequence of runnables on the job's resources. You can configure a job's tasks to run in parallel or sequentially.

  • Task: one execution of a job's sequence of runnables. When a job has multiple tasks, you specify how you want each execution to vary by referencing the environment variable for a task's index in the job's runnables.

  • Runnable: an executable script or container that you define as part of a job.

  • Resources: the infrastructure needed to run a job. At minimum, you only need to specify the compute resources required per task: CPU cores, memory, and (if required) extra boot disk storage. Optionally, you can also specify other resource options for the job. Batch automatically creates and deletes the resources that meet your specifications during the job's run time.

    Specifically, each Batch job runs on a regional managed instance group (MIG), which is a group of one or more matching Compute Engine virtual machine (VM) instances that are each located in one of the included zones. Each VM has dedicated hardware for CPU cores and memory—which affect the performance of your job—and a boot disk—which stores an operating system (OS) image and instructions for running your job. If specified, a job might also include or access additional resources—for example, GPUs or external storage volumes. The number of VMs provisioned for a job depends on the hardware requirements and task scheduling options that you specify.

In summary, Batch lets you create and run jobs that each automatically provision and utilize the resources required to execute its tasks.

For more information about job behavior and options, see the Job creation and execution overview page.

Pricing

There is no additional cost for using Batch. You are only charged for the cost of the underlying resources required to execute your jobs.

For more information about the costs associated with Batch and how to filter Cloud Billing reports to view Batch costs, see Pricing.

Restrictions

Batch has the following restrictions:

  • You can't exceed the Batch quotas and limits for your project.
  • Jobs can only use machine types from the following machine series:
    • General-purpose: C4, C4A, C4D, C3, C3D, E2, N4, N4A, N2D, N2, N1, and T2D
    • Compute-optimized: C2, C2D, and H3
    • Memory-optimized: M3, M2, and M1
    • Accelerator-optimized: A4, A3 Ultra, A3 Mega, A3 High, A2, G4, and G2
  • You can't specify more than one task group per job. All jobs have only one task group named group0.

Prerequisites

To start using Batch, complete the following prerequisites:

  1. If your project has not used Batch before, enable Batch for your project.
  2. Set up Batch for each new user.

Enable Batch for a project

To start using Batch with a project, do the following:

  1. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Roles required to select or create a project

    • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
    • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

    Go to project selector

  2. Verify that billing is enabled for your Google Cloud project.