databricks

Databricks SDK for Go

Beta: This SDK is supported for production use cases, but we do expect future releases to have some interface changes; see Interface stability. We are keen to hear feedback from you on these SDKs. Please file issues, and we will address them | See documentation at Go Packages | See also the Terraform Provider | See also the SDK for Python | See also the SDK for Java

The Databricks SDK for Go includes functionality to accelerate development with Go for the Databricks Lakehouse. It covers all public Databricks REST API operations. The SDK's internal HTTP client is robust and handles failures on different levels by performing intelligent retries.

Contents

• Databricks SDK for Go
  • Contents
  • Getting started
  • Authentication
    • In this section
    • Default authentication flow
    • Unified host support
    • Databricks native authentication
    • Azure native authentication
    • Google Cloud Platform native authentication
    • Overriding .databrickscfg
    • Additional authentication configuration options
    • Custom credentials provider
  • Code examples
  • Long-running operations
    • In this section
    • Command execution on clusters
    • Cluster library management
    • Advanced usage
  • Paginated responses
  • Retries
  • GetByName utility methods
  • Node type and Databricks Runtime selectors
  • Integration with io interfaces for DBFS
    • Reading into and writing from buffers
  • pflag.Value for enums
  • User Agent Request Attribution
  • Error handling
  • Logging
• Testing
  • Interface stability
  • Go version support

Getting started

1. On your local development machine with Go already installed and a Go code project active, create a go.mod file to track your Go code's dependencies by running the go mod init command, for example:

   go mod init sample
   

2. Take a dependency on the Databricks SDK for Go package by running the go mod edit -require command:

   go mod edit -require github.com/databricks/databricks-sdk-go@latest
   

   Your go.mod file should now look like this:

   module sample
   
   go 1.25
   
   require github.com/databricks/databricks-sdk-go v0.9.0
   
   // Indirect dependencies will go here.
   

3. Within your project, create a Go code file that imports the Databricks SDK for Go. The following example, in a file named main.go with the following contents, simply lists all the clusters in your Databricks workspace:

   package main
   
   import (
     "context"
   
     "github.com/databricks/databricks-sdk-go"
     "github.com/databricks/databricks-sdk-go/service/compute"
   )
   
   func main() {
     w := databricks.Must(databricks.NewWorkspaceClient())
     all, err := w.Clusters.ListAll(context.Background(), compute.ListClustersRequest{})
     if err != nil {
       panic(err)
     }
     for _, c := range all {
       println(c.ClusterName)
     }
   }
   

4. Add any misssing module dependencies by running the go mod tidy command:

   go mod tidy
   

   Note: If you get the error go: warning: "all" matched no packages, you forgot to add the preceding Go code file that imports the Databricks SDK for Go.

5. Grab copies of all packages needed to support builds and tests of packages in your main module, by running the go mod vendor command:

   go mod vendor
   

6. Set up Databricks authentication on your local development machine by running databricks configure command, if you have not done so already. For details, see the next section, Authentication.

7. Run your Go code file, assuming a file named main.go, by running the go run command:

   go run main.go
   

   Assuming the preceding example code is run, the output is:

   [TRACE] Loading config via environment
   [TRACE] Loading config via config-file
   ...
   [TRACE] Attempting to configure auth: pat
   [TRACE] Attempting to configure auth: basic
   [TRACE] Attempting to configure auth: azure-client-secret
   ...
   

Authentication

If you use Databricks configuration profiles or Databricks-specific environment variables for Databricks authentication, the only code required to start working with a Databricks workspace is the following code snippet, which instructs the Databricks SDK for Go to use its default authentication flow:

w := databricks.Must(databricks.NewWorkspaceClient())
w./*press TAB for autocompletion*/

The conventional name for the variable that holds the workspace-level client of the Databricks SDK for Go is w, which is shorthand for workspace.

In this section

• Default authentication flow
• Unified host support
• Databricks native authentication
• Azure native authentication
• Google Cloud Platform native authentication
• Overriding .databrickscfg
• Additional authentication configuration options
• Custom credentials provider

Default authentication flow

If you run the Databricks Terraform Provider, the Databricks CLI, or applications that target the Databricks SDKs for other langauges, most likely they will all interoperate nicely together. By default, the Databricks SDK for Go tries the following authentication methods, in the following order, until it succeeds:

1. Databricks native authentication
2. Azure native authentication
3. Google Cloud Platform native authentication
4. If the SDK is unsuccessful at this point, it returns an authentication error and stops running.

Each authentication method requires specific configuration attributes (e.g., Token for PAT auth, AzureClientID for Azure service principal auth). The SDK automatically detects the cloud provider and skips authentication methods whose required configuration attributes are not present. This means that Azure-specific methods like azure-cli are automatically skipped when connecting to an AWS or GCP workspace, and vice versa for GCP-specific methods.

To force a specific authentication method instead of relying on auto-detection, set the AuthType field in *databricks.Config:

// Force Azure CLI authentication — skip all other methods
w, err := databricks.NewWorkspaceClient(&databricks.Config{
    Host:     "https://mycompany.databricks.com",
    AuthType: "azure-cli",
    Cloud:    "Azure",
})

This is useful when your environment has credentials for multiple authentication methods and you want to ensure a specific one is used or when auto detection is not accurate.

For each authentication method, the SDK searches for compatible authentication credentials in the following locations, in the following order. Once the SDK finds a compatible set of credentials that it can use, it stops searching:

1. Credentials that hard-coded into *databricks.Config.

   Caution: Databricks does not recommend hard-coding credentials into *databricks.Config, as they can be exposed in plain text in version control systems. Use environment variables or configuration profiles instead.

2. Credentials in Databricks-specific environment variables.

3. For Databricks native authentication, credentials in the .databrickscfg file's DEFAULT configuration profile from its default file location (~ for Linux or macOS, and %USERPROFILE% for Windows).

4. For Azure or Google Cloud Platform native authentication, the SDK searches for credentials through the Azure CLI or Google Cloud CLI as needed.

Depending on the Databricks authentication method, the SDK uses the following information. Presented are the *databricks.Config arguments, their descriptions, any corresponding environment variables, and any corresponding .databrickscfg file fields, respectively.

Unified host support

Certain Databricks host types support both account-level and workspace-level API operations from a single endpoint. When using such a unified host, a single configuration profile can be used to create both WorkspaceClient and AccountClient instances without changing the Host.

For this to work, the following conditions must be met:

1. The host must support unified operations.
2. Both AccountID and WorkspaceID must be available — either set explicitly in the configuration or auto-discovered.

When both values are present, the SDK uses WorkspaceID to route workspace-level requests and AccountID to route account-level requests, all through the same host.

# .databrickscfg
[unified]
host         = https://mycompany.databricks.com
account_id   = 00000000-0000-0000-0000-000000000000
workspace_id = 1234567890

// Both clients share the same host and profile
w, _ := databricks.NewWorkspaceClient(&databricks.Config{Profile: "unified"})
a, _ := databricks.NewAccountClient(&databricks.Config{Profile: "unified"})

// A WorkspaceClient for a different workspace under the same host and account
w, _ = databricks.NewWorkspaceClient(&databricks.Config{
    Profile:     "unified",
    WorkspaceID: "2345678901",
})

If the host supports it, AccountID and WorkspaceID may be auto-discovered, reducing the required explicit configuration.

Databricks native authentication

By default, the Databricks SDK for Go initially tries Databricks token authentication (AuthType: "pat" in *databricks.Config). If the SDK is unsuccessful, it then tries Workload Identity Federation (WIF). See Supported WIF for the supported JWT token providers.

• For Databricks token authentication, you must provide Host and Token; or their environment variable or .databrickscfg file field equivalents.
• For Databricks OIDC authentication, you must provide the Host, ClientId and TokenAudience (optional) either directly, through the corresponding environment variables, or in your .databrickscfg configuration file. More information can be found in Databricks Documentation
• For Azure DevOps OIDC authentication, the TokenAudience is irrelevant as the audience is always set to api://AzureADTokenExchange. Also, the System.AccessToken pipeline variable required for OIDC request must be exposed as the SYSTEM_ACCESSTOKEN environment variable, following Pipeline variables

During initialization, the SDK automatically resolves missing configuration fields (AccountID, WorkspaceID, Cloud, and DiscoveryURL). Any explicitly provided values take precedence and are never overwritten. If the auto discovery fails, the SDK falls back to the explicit configuration. It is recommended to always set explicit configuration.