Migrate from AWS Step Functions to Workflows

To help you prepare to migrate from Amazon Web Services (AWS) Step Functions to Workflows on Google Cloud, this page explains key similarities and differences between the two products. This information is intended to assist those who are already familiar with Step Functions implement a similar architecture using Workflows.

Like Step Functions, Workflows is a fully managed, state-based orchestration platform that executes services in an order that you define: a workflow. These workflows can combine services including custom services hosted on Cloud Run or Cloud Run functions, Google Cloud services such as Cloud Vision AI and BigQuery, and any HTTP-based API.

Note that Step Functions Express Workflows is an AWS Step Functions workflow type that is not considered here since the duration of an Express Workflow is limited, and an exactly-once workflow execution is not supported.

Hello world

In Step Functions, a state machine is a workflow, and a task is a state in a workflow that represents a single unit of work that another AWS service performs. Step Functions requires every state to define the next state.

In Workflows, a series of steps using the Workflows syntax describes the tasks to execute. Workflows treats steps as if they are in an ordered list and executes them one at a time until all the steps have run.

The following "Hello world" sample demonstrates the use of states in Step Functions and steps in Workflows:

  {
    "Comment": "Hello world example of Pass states in Amazon States Language",
    "StartAt": "Hello",
    "States": {
      "Hello": {
        "Type": "Pass",
        "Result": "Hello",
        "Next": "World"
      },
      "World": {
        "Type": "Pass",
        "Result": "World",
        "End": true
      }
    }
  }

  ---
  # Hello world example of steps using Google Cloud Workflows syntax
  main:
      steps:
      - Hello:
          next: World
      - World:
          next: end
  ...

  {
    "main": {
      "steps": [
        {
          "Hello": {
            "next": "World"
          }
        },
        {
          "World": {
            "next": "end"
          }
        }
      ]
    }
  }

Comparison overview

This section compares the two products in more detail.

Syntax

Step Functions primarily uses JSON to define functions and does not support YAML directly; however, in the AWS Toolkit for Visual Studio Code and in AWS CloudFormation, you can use YAML for a Step Functions definition.

You can describe Workflows steps using the Workflows syntax, and they can be written in either YAML or JSON. The majority of workflows are in YAML. The examples on this page demonstrate the advantages of YAML, including ease of reading and writing, and native support of comments. For a detailed explanation of the Workflows syntax, see the Syntax reference.

Control flow

Both Workflows and Step Functions model workflows as a series of tasks: steps in Workflows and states in Step Functions. Both allow a task to indicate which task to execute next and support switch-like conditionals to pick the next unit of work based on the current state. One key difference is that Step Functions requires every state to define the next one, while Workflows executes steps in the order that they are specified (including alternative next steps). For more information, see Conditions and Steps.

Worker

Both products orchestrate compute resources such as functions, containers, and other web services to get things done. In Step Functions, the worker is identified by a Resource field which is syntactically a URI. The URIs used to identify worker resources are in Amazon Resource Name (ARN) format. To directly invoke an arbitrary HTTP endpoint, you can define an HTTP task.

Workflows can send HTTP requests to an arbitrary HTTP endpoint and get a response. Connectors make it easier to connect to other Google Cloud APIs within a workflow, and to integrate your workflows with other Google Cloud products like Pub/Sub, BigQuery, or Cloud Build. Connectors simplify calling services because they handle the formatting of requests for you, providing methods and arguments so that you don't need to know the details of a Google Cloud API. You can also configure the timeout and polling policies.

Reliability

If a task fails, your workflow must be able to retry appropriately, catch exceptions when it has to, and reroute the workflow as necessary. Both Step Functions and Workflows achieve reliability using similar mechanisms: exception catching with retries, and eventual dispatching to elsewhere in the workflow. For more information, see Workflow errors.

Parallelism

You might want your workflow to orchestrate multiple tasks in parallel. Step Functions provides two ways to achieve this: you can take one data item and pass it in parallel to multiple different tasks; or, you can use an array and pass its elements to the same task.

In Workflows, you can define a part of your workflow where two or more steps can execute concurrently. You can define either branches that run concurrently, or a loop where iterations run concurrently. For details, see Execute workflow steps in parallel.

State data

One benefit of a workflow engine is that state data is maintained for you without an external datastore. In Step Functions, the state data is passed along in a JSON structure from one state to another.

In Workflows, you can save the state data in global variables. Since you are allowed up to a one-year execution duration, you can keep the state data as long as the instance is still in execution.

Authentication

Both products rely upon an underlying Identity and Access Management (IAM) system for authentication and access control. For example, you can use an IAM role to invoke Step Functions.

In Workflows, you can use a service account to invoke a workflow; you can use OAuth 2.0 or OIDC to connect with Google Cloud APIs; and you can use an authorization request header to authenticate with a third-party API. For more information, see Grant a workflow permission to access Google Cloud resources and Make authenticated requests from a workflow.

User experience

You can use a command-line tool, or infrastructure as code (IaC) such as Terraform, to define and manage both Step Functions and Workflows.

In addition, Workflows supports the execution of workflows using the client libraries, in the Google Cloud console, using the Google Cloud CLI, or by sending a request to the Workflows REST API. For details, see Execute a workflow.

Pricing

Both products have a free tier. For more details, see their respective pricing pages: Step Functions pricing and Workflows pricing.

Mapping state types to steps

There are eight state types in Step Functions. States are elements in a state machine that can make decisions based on their input, perform actions, and pass output to other states. Before migrating from Step Functions to Workflows, ensure that you understand how to translate each state type to a Workflows step.

Choice

A Choice state adds branching logic to a state machine.

In Workflows, you can use a switch block as a selection mechanism that allows the value of an expression to control the flow of a workflow's execution. If a value matches, that condition's statement is executed. For more information, see Conditions.

  "ChoiceState": {
    "Type": "Choice",
    "Choices": [
      {
        "Variable": "$.foo",
        "NumericEquals": 1,
        "Next": "FirstMatchState"
      },
      {
        "Variable": "$.foo",
        "NumericEquals": 2,
        "Next": "SecondMatchState"
      }
    ],
    "Default": "DefaultState"
  }

  switch:
    - condition: ${result.body.SomeField < 10}
      next: small
    - condition: ${result.body.SomeField < 100}
      next: medium
    - condition: true
      next: default_step

  {
    "switch": [
      {
        "condition": "${result.body.SomeField < 10}",
        "next": "small"
      },
      {
        "condition": "${result.body.SomeField < 100}",
        "next": "medium"
      },
      {
        "condition": true,
        "next": "default_step"
      }
    ]
  }

Fail

A Fail state stops the execution of the state machine and marks it as a failure.

In Workflows, you can raise custom errors using the raise syntax, and you can catch and handle errors using a try/except block. For more information, see Raise errors.

  "FailState": {
      "Type": "Fail",
      "Error": "ErrorA",
      "Cause": "Kaiju attack"
  }

  raise:
      code: 55
      message: "Something went wrong."

  {
    "raise": {
      "code": 55,
      "message": "Something went wrong."
    }
  }

Map

A Map state can be used to run a set of steps for each element of an input array.

In Workflows, you can use a for loops for iterations.

  { "StartAt": "ExampleMapState",
    "States": {
      "ExampleMapState": {
        "Type": "Map",
        "Iterator": {
           "StartAt": "CallLambda",
           "States": {
             "CallLambda": {
               "Type": "Task",
               "Resource": "arn:aws:lambda:us-east-1:123456789012:function:HelloFunction",
               "End": true
             }
           }
        }, "End": true
      }
    }
  }

  - assignStep:
      assign:
        - map:
            1: 10
            2: 20
            3: 30
        - sum: 0
  - loopStep:
      for:
          value: key
          in: ${keys(map)}
          steps:
            - sumStep:
                assign:
                  - sum: ${sum + map[key]}
  - returnStep:
      return: ${sum}

  [
    {
      "assignStep": {
        "assign": [
          {
            "map": {
              "1": 10,
              "2": 20,
              "3": 30
            }
          },
          {
            "sum": 0
          }
        ]
      }
    },
    {
      "loopStep": {
        "for": {
          "value": "key",
          "in": "${keys(map)}",
          "steps": [
            {
              "sumStep": {
                "assign": [
                  {
                    "sum": "${sum + map[key]}"
                  }
                ]
              }
            }
          ]
        }
      }
    },
    {
      "returnStep": {
        "return": "${sum}"
      }
    }
  ]

Parallel

A Parallel state can be used to create parallel branches of execution in your state machine. In the following Step Functions example, an address and phone lookup is performed in parallel.

In Workflows, you can use a parallel step to define a part of your workflow where two or more steps can execute concurrently. For more information, see Parallel steps.

  { "StartAt": "LookupCustomerInfo",
    "States": {
      "LookupCustomerInfo": {
        "Type": "Parallel",
        "End": true,
        "Branches": [
          {
           "StartAt": "LookupAddress",
           "States": {
             "LookupAddress": {
               "Type": "Task",
               "Resource": "arn:aws:lambda:us-east-1:123456789012:function:AddressFinder",
               "End": true
             }
           }
         },
         {
           "StartAt": "LookupPhone",
           "States": {
             "LookupPhone": {
               "Type": "Task",
               "Resource": "arn:aws:lambda:us-east-1:123456789012:function:PhoneFinder",
               "End": true
             }
           }
         }
        ]
      }
    }
  }

  main:
     params: [args]
     steps:
     - init:
         assign:
         - workflow_id: "lookupAddress"
         - customer_to_lookup:
             - address: ${args.customerId}
             - phone: ${args.customerId}
         - addressed: ["", ""] # to write to this variable, you must share it
     - parallel_address:
         parallel:
             shared: [addressed]
             for:
                 in: ${customer_to_lookup}
                 index: i # optional, use if index is required
                 value: arg
                 steps:
                 - address:
                     call: googleapis.workflowexecutions.v1.projects.locations.workflows.executions.run
                     args:
                         workflow_id: ${workflow_id}
                         argument: ${arg}
                     result: r
                 - set_result:
                     assign:
                     - addressed[i]: ${r}
     - return:
             return: ${addressed}

  {
    "main": {
      "params": [
        "args"
      ],
      "steps": [
        {
          "init": {
            "assign": [
              {
                "workflow_id": "lookupAddress"
              },
              {
                "customer_to_lookup": [
                  {
                    "address": "${args.customerId}"
                  },
                  {
                    "phone": "${args.customerId}"
                  }
                ]
              },
              {
                "addressed": [
                  "",
                  ""
                ]
              }
            ]
          }
        },
        {
          "parallel_address": {
            "parallel": {
              "shared": [
                "addressed"
              ],
              "for": {
                "in": "${customer_to_lookup}",
                "index": "i",
                "value": "arg",
                "steps": [
                  {
                    "address": {
                      "call": "googleapis.workflowexecutions.v1.projects.locations.workflows.executions.run",
                      "args": {
                        "workflow_id": "${workflow_id}",
                        "argument": "${arg}"
                      },
                      "result": "r"
                    }
                  },
                  {
                    "set_result": {
                      "assign": [
                        {
                          "addressed[i]": "${r}"
                        }
                      ]
                    }
                  }
                ]
              }
            }
          }
        },
        {
          "return": {
            "return": "${addressed}"
          }
        }
      ]
    }
  }

Pass

A Pass state passes its input to its output, without performing work. This is commonly used to manipulate the state data in the JSON.

As Workflows does not pass along data in this way, you can leave the state as a no-op, or you can use an assign step to modify the variables. For more information, see Assign variables.

  "No-op": {
    "Type": "Pass",
    "Result": {
      "x-datum": 0.38,
      "y-datum": 622.22
    },
    "ResultPath": "$.coords",
    "Next": "End"
  }

  assign:
      - number: 5
      - number_plus_one: ${number+1}
      - other_number: 10
      - string: "hello"

  {
    "assign": [
      {
        "number": 5
      },
      {
        "number_plus_one": "${number+1}"
      },
      {
        "other_number": 10
      },
      {
        "string": "hello"
      }
    ]
  }

Succeed

A Succeed state stops an execution successfully.

In Workflows, you can use return in the main workflow to stop a workflow's execution. You can also finish a workflow by completing the final step (assuming that the step doesn't jump to another), or you can use next: end to stop a workflow's execution if you don't need to return a value. For more information, see Complete the execution of a workflow.

  "SuccessState": {
    "Type": "Succeed"
  }

  return: "Success!"
  next: end

  {
    "return": "Success!",
    "next": "end"
  }

Task

A Task state represents a single unit of work performed by a state machine. In the following Step Functions example, it invokes a Lambda function. (Activities are an AWS Step Functions feature that enables you to have a task in your state machine where the work is performed elsewhere.)

In the Workflows example, a call is made to an HTTP endpoint to invoke a Cloud Run function. You can also use a connector which allows easy access to other Google Cloud products. Additionally, you can pause a workflow and poll for data. Or, you can use a callback endpoint to signal to your workflow that a specified event has occurred, and wait on that event without polling.

  "HelloWorld": {
    "Type": "Task",
    "Resource": "arn:aws:lambda:us-east-1:123456789012:function:HelloFunction",