Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Important
This section doesn't apply to compute that manages dependencies with environments. On that compute, you manage dependencies with base environments instead of cluster libraries.
To make third-party or custom code available to notebooks and jobs running on your compute resources, you can install a library. Libraries can be written in Python, Java, Scala, and R. You can upload Python, Java, and Scala libraries and point to external packages in PyPI, Maven, and CRAN repositories.
Azure Databricks includes many common libraries in Databricks Runtime. To see which libraries are included in Databricks Runtime, look at the System Environment subsection of the Databricks Runtime release notes for your Databricks Runtime version.
Note
Microsoft Support helps isolate and resolve issues related to libraries installed and maintained by Azure Databricks. For third-party components, including libraries, Microsoft provides commercially reasonable support to help you further troubleshoot issues. Microsoft Support assists on a best-effort basis and might be able to resolve the issue. For open source connectors and projects hosted on Github, we recommend that you file issues on Github and follow up on them. Development efforts such as shading jars or building Python libraries are not supported through the standard support case submission process: they require a consulting engagement for faster resolution. Support might ask you to engage other channels for open-source technologies where you can find deep expertise for that technology. There are several community sites; two examples are the Microsoft Q&A page for Azure Databricks and Stack Overflow.
Compute-scoped libraries
You can install libraries on a compute resource so that they can be used by all notebooks and jobs running on the compute. Databricks supports Python, JAR, and R libraries. See Compute-scoped libraries.
You can install a compute-scoped library directly from the following sources:
- A package repository such as PyPI, Maven, or CRAN
- Workspace files
- Unity Catalog volumes
- A cloud object storage location
- A path on your local machine
Not all locations are supported for all types of libraries or all compute configurations. See Recommendations for uploading libraries for configuration recommendations.
Important
Libraries can be installed from DBFS when using Databricks Runtime 14.3 LTS and below. However, any workspace user can modify library files stored in DBFS. To improve the security of libraries in a Azure Databricks workspace, storing library files in the DBFS root is deprecated and disabled by default in Databricks Runtime 15.1 and above. See Storing libraries in DBFS root is deprecated and disabled by default.
Instead, Databricks recommends uploading all libraries, including Python libraries, JAR files, and Spark connectors, to workspace files or Unity Catalog volumes, or using library package repositories. If your workload does not support these patterns, you can also use libraries stored in cloud object storage.
For complete library support information, see Python library support, Java and Scala library support, and R library support.
Recommendations for uploading libraries
Databricks supports most configuration installations of Python, JAR, and R libraries, but there are some unsupported scenarios. It is recommended that you upload libraries to source locations that support installation onto compute with standard access mode (formerly shared access mode), as this is the recommended mode for all workloads. See Access modes. When scheduling jobs with standard access mode run the job with a service principal.
Important
Only use compute with dedicated access mode (formerly single user access mode) if required functionality is not supported by standard access mode. No isolation shared access mode is a legacy configuration on Databricks that is not recommended.
The following table provides recommendations organized by Databricks Runtime version and Unity Catalog enablement.
| Configuration | Recommendation |
|---|---|
| Databricks Runtime 13.3 LTS and above with Unity Catalog | Install libraries on compute with standard access mode from Unity Catalog volumes with GRANT READ for necessary users. If applicable, Maven coordinates and JAR library paths need to be added to the allowlist. |
| Databricks Runtime 11.3 LTS and above without Unity Catalog | Install libraries from workspace files. (File size limit is 500 MB.) |
| Databricks Runtime 10.4 LTS and below | Install libraries from cloud object storage. |
Python library support
The following table indicates Databricks Runtime version compatibility for Python wheel files for different compute access modes based on the library source location. See Databricks Runtime release notes versions and compatibility and Access modes.
In Databricks Runtime 15.0 and above, you can use requirements.txt files to manage your Python dependencies. These files can be uploaded to any supported source location.
Note
Installing Python egg files is only supported on Databricks Runtime 13.3 LTS and below, and only for dedicated or no isolation shared access modes. In addition, you cannot install Python egg files on volumes or workspace files. Use Python wheel files or install packages from PyPI instead.
| Standard access mode | Dedicated access mode | No isolation shared access mode (Legacy) | |
|---|---|---|---|