Create and run an example job

Learn how to create and run an example batch processing job that transcodes videos by using Batch for Google Cloud.

To follow step-by-step guidance for this task directly in the Google Cloud console, click Guide me:

Before you begin

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

Install the Google Cloud CLI.

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

To initialize the gcloud CLI, run the following command:

gcloud init

Create or select a Google Cloud project.

Roles required to select or create a project

Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Create a Google Cloud project:
```
gcloud projects create PROJECT_ID
```
Replace PROJECT_ID with a name for the Google Cloud project you are creating.
Select the Google Cloud project that you created:
```
gcloud config set project PROJECT_ID
```
Replace PROJECT_ID with your Google Cloud project name.

Verify that billing is enabled for your Google Cloud project.

Enable the Batch, Compute Engine, Logging and Cloud Storage APIs:

Roles required to enable APIs

To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

gcloud services enable batch.googleapis.com compute.googleapis.com logging.googleapis.com storage.googleapis.com

Install the Google Cloud CLI.

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

To initialize the gcloud CLI, run the following command:

gcloud init

Create or select a Google Cloud project.

Roles required to select or create a project

Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Create a Google Cloud project:
```
gcloud projects create PROJECT_ID
```
Replace PROJECT_ID with a name for the Google Cloud project you are creating.
Select the Google Cloud project that you created:
```
gcloud config set project PROJECT_ID
```
Replace PROJECT_ID with your Google Cloud project name.

Verify that billing is enabled for your Google Cloud project.

Enable the Batch, Compute Engine, Logging and Cloud Storage APIs:

Roles required to enable APIs

gcloud services enable batch.googleapis.com compute.googleapis.com logging.googleapis.com storage.googleapis.com

Make sure that you and the job's service account have the required permissions to complete this tutorial. This tutorial uses default service account for a job, which is the Compute Engine default service account.
Note: If you created a new project, this step isn't required:
- As the project creator, you're automatically granted the Owner (roles/owner) IAM role, which already has all the permissions required for this tutorial.
- In a new project, the Compute Engine default service account is automatically granted the Editor role (roles/editor), which already has all the permissions required for this tutorial.
- To get the permissions that you need to complete this tutorial, ask your administrator to grant you the following IAM roles:
  - To create, view, and delete jobs:
    - Batch Job Editor (roles/batch.jobsEditor) on the project
    - Service Account User (roles/iam.serviceAccountUser) on the Compute Engine default service account
  - To create, view, and delete Cloud Storage buckets: Storage Admin (roles/storage.admin) on the project
  - To view logs from jobs: Logs Viewer (roles/logging.viewer) on the project
  For more information about granting roles, see Manage access to projects, folders, and organizations.
  
  You might also be able to get the required permissions through custom roles or other predefined roles.
- To ensure that the Compute Engine default service account has the necessary permissions to complete this tutorial, ask your administrator to grant the following IAM roles to the Compute Engine default service account:
  
  Important: You must grant these roles to the Compute Engine default service account, not to your user account. Failure to grant the roles to the correct principal might result in permission errors.
  - Batch Agent Reporter (roles/batch.agentReporter) on the project
  - To let jobs access Cloud Storage buckets: Storage Admin (roles/storage.admin) on the project
  - To let jobs generate logs in Logging: Logs Writer (roles/logging.logWriter) on the project

Clone the Batch git repository into the current directory:

git clone https://github.com/GoogleCloudPlatform/batch-samples.git

Go to the transcoding directory:
```
cd batch-samples/transcoding/
```

Prepare job inputs

Create a Cloud Storage bucket:
```
gcloud storage buckets create gs://BUCKET_NAME
```
Replace BUCKET_NAME with a globally unique name for the bucket.

Caution: Don't include sensitive information in a Cloud Storage bucket name. A bucket name is global and publicly visible.

The output is similar to the following:
```
Creating gs://BUCKET_NAME/...
```

Copy the transcode.sh script and the folder containing the video files to your Cloud Storage bucket:

gcloud storage cp -R transcode.sh input gs://BUCKET_NAME

The output is similar to the following:

Copying file://transcode.sh to gs://BUCKET_NAME/transcode.sh
Copying file://input/video-2.mp4 to gs://BUCKET_NAME/input/video-2.mp4
Copying file://input/video-1.mp4 to gs://BUCKET_NAME/input/video-1.mp4
Copying file://input/video-0.mp4 to gs://BUCKET_NAME/input/video-0.mp4
  Completed files 4/4 | 37.5MiB/37.5MiB

Average throughput: 48.4MiB/s

Create a job

In a text editor of your choice, open the job.json configuration file.

Set the value of the remotePath field to the name of your Cloud Storage bucket:

{
  "taskGroups": [
    {
      "taskSpec": {
        "runnables": [
          {
            "script": {
              "text": "bash /mnt/share/transcode.sh"
            }
          }
        ],
        "computeResource": {
          "cpuMilli": 2000,
          "memoryMib": 2048
        },
        "volumes": [
          {
            "gcs": {
              "remotePath": "BUCKET_NAME"
            },
            "mountPath": "/mnt/share"