Orchestrate jobs by running Nextflow pipelines on Batch

This tutorial explains how to run a Nextflow pipeline on Batch. Specifically, this tutorial runs the sample rnaseq-nf life sciences pipeline from Nextflow, which quantifies genomic features from short read data using RNA-Seq.

This tutorial is intended for Batch users who want to use Nextflow with Batch.

Nextflow is open-source software for orchestrating bioinformatics workflows.

Objectives

By completing this tutorial, you'll learn how to do the following:

Install Nextflow in Cloud Shell.
Create a Cloud Storage bucket.
Configure a Nextflow pipeline.
Run a sample pipeline using Nextflow on Batch.
View outputs of the pipeline.
Clean up to avoid incurring additional charges by doing one of the following:
- Delete a project.
- Delete individual resources.

Costs

In this document, you use the following billable components of Google Cloud:

Batch
Cloud Storage

To generate a cost estimate based on your projected usage, use the pricing calculator.

New Google Cloud users might be eligible for a free trial.

The resources created in this tutorial typically cost less than a dollar, assuming you complete all the steps—including the cleanup—in a timely manner.

Before you begin

Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

Install the Google Cloud CLI.

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

To initialize the gcloud CLI, run the following command:

gcloud init

Create or select a Google Cloud project.

Roles required to select or create a project

Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Create a Google Cloud project:
```
gcloud projects create PROJECT_ID
```
Replace PROJECT_ID with a name for the Google Cloud project you are creating.
Select the Google Cloud project that you created:
```
gcloud config set project PROJECT_ID
```
Replace PROJECT_ID with your Google Cloud project name.

Verify that billing is enabled for your Google Cloud project.

Enable the Batch, Cloud Storage, Compute Engine, and Logging APIs:

Roles required to enable APIs

To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

gcloud services enable batch.googleapis.com compute.googleapis.com logging.googleapis.com storage.googleapis.com

Install the Google Cloud CLI.

If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

To initialize the gcloud CLI, run the following command:

gcloud init

Create or select a Google Cloud project.

Roles required to select or create a project

Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

Create a Google Cloud project:
```
gcloud projects create PROJECT_ID
```
Replace PROJECT_ID with a name for the Google Cloud project you are creating.
Select the Google Cloud project that you created:
```
gcloud config set project PROJECT_ID
```
Replace PROJECT_ID with your Google Cloud project name.

Verify that billing is enabled for your Google Cloud project.

Enable the Batch, Cloud Storage, Compute Engine, and Logging APIs:

Roles required to enable APIs

gcloud services enable batch.googleapis.com compute.googleapis.com logging.googleapis.com storage.googleapis.com

Make sure that your project has a Virtual Private Cloud (VPC) network with a valid networking configuration for this tutorial.

Note: No action is required for this step unless your organization disables default network creation by enabling the compute.skipDefaultNetworkCreation policy constraint or your project has modified its default network.

This tutorial assumes that you are using the default network. By default, Google Cloud resources use the default network, which provides the network access required for this tutorial.

Objective: Optionally, if you want a Batch job for a Nextflow pipeline to use a different network, you can modify the nextflow.config file in this tutorial to also define the job's network and subnet by using the google.batch.network and google.batch.subnetwork Nextflow fields, respectively. However, you must also make sure that the network and subnet are correctly configured for your workload. For more information, see Batch networking overview.

Make sure that your project has at least one service account with the permissions required for running the Batch job in this tutorial.

By default, jobs use the Compute Engine default service account, which is automatically granted the Editor (roles/editor) IAM role and already has all the permissions required for this tutorial.

Note: No action is required for this step unless your organization disables automatic permissions for default service accounts by enabling the iam.automaticIamGrantsForDefaultServiceAccounts policy constraint, your project has modified its Compute Engine default service account, or you want to use a different service account.

To ensure that the job's service account has the necessary permissions to allow the Batch service agent to create and access resources for Batch jobs, ask your administrator to grant the following IAM roles to the job's service account:

Important: You must grant these roles to the job's service account, not to your user account. Failure to grant the roles to the correct principal might result in permission errors.
- Batch Agent Reporter (roles/batch.agentReporter) on the project
- Storage Admin (roles/storage.admin) on the project
- (Recommended) Let jobs generate logs in Cloud Logging: Logs Writer (roles/logging.logWriter) on the project
For more information about granting roles, see Manage access to projects, folders, and organizations.

Your administrator might also be able to give the job's service account the required permissions through custom roles or other predefined roles.

Make sure that you have the permissions required for this tutorial.

Note: If you created the project that you plan to use for this tutorial, no action is required for this step. As the project creator, you are automatically granted the Owner (roles/owner) IAM role, which already has all the permissions required for this tutorial.

To get the permissions that you need to complete this tutorial, ask your administrator to grant you the following IAM roles:
- Batch Job Editor (roles/batch.jobsEditor) on the project
- Service Account User (roles/iam.serviceAccountUser) on the job's service account
- Storage Object Admin (roles/storage.objectAdmin) on the project

Install Nextflow:

curl -s -L https://github.com/nextflow-io/nextflow/releases/download/v23.04.1/nextflow | bash

The output should be similar to the following:

N E X T F L O W
version 23.04.1 build 5866
created 15-04-2023 06:51 UTC
cite doi:10.1038/nbt.3820
http://nextflow.io

Nextflow installation completed. Please note:
- the executable file `nextflow` has been created in the folder: ...
- you may complete the installation by moving it to a directory in your $PATH

Create a Cloud Storage bucket

To create a Cloud Storage bucket to store temporary work and output files from the Nextflow pipeline, use the Google Cloud console or the command-line.

Console

To create a Cloud Storage bucket using the Google Cloud console, follow these steps:

In the Google Cloud console, go to the Buckets page.

Go to Buckets
Click Create.
On the Create a bucket page, enter a globally unique name for your bucket.
Click Create.
In the Public access will be prevented window, click Confirm.

gcloud

To create a Cloud Storage bucket using the Google Cloud CLI, use the gcloud storage buckets create command.

gcloud storage buckets create gs://BUCKET_NAME

Replace BUCKET_NAME with a globally unique name for your bucket.

If the request is successful, the output should be similar to the following:

Creating gs://BUCKET_NAME/...
   ```

Configure Nextflow

To configure the Nextflow pipeline to run on Batch, follow these steps in the command-line:

Clone the sample pipeline repository:

git clone https://github.com/nextflow-io/rnaseq-nf.git

Go to the rnaseq-nf folder:
```
cd rnaseq-nf
```

Open the nextflow.config file:

nano nextflow.config

The file should contain the following google-batch section:

'google-batch' {
    params.transcriptome = 'gs://rnaseq-nf/data/ggal/transcript.fa'
    params.reads = 'gs://rnaseq-nf/data/ggal/gut_{1,2}.fq'
    params.multiqc = 'gs://rnaseq-nf/multiqc'
    process.executor = 'google-batch'
    process.container = 'docker.io/nextflow/rnaseq-nf:v1.3.1'
    /*
     * replace with your own bucket!
     */
    workDir = 'gs://BUCKET_NAME/WORK_DIRECTORY'
    google.region  = 'REGION'
}

In the google-batch section, do the following:
1. Replace BUCKET_NAME with the name of the Cloud Storage bucket you created in the previous steps.
2. Replace WORK_DIRECTORY with the name for a new folder that the pipeline can use to store logs and outputs.
  
  For example, enter workDir.
3. Replace