Libraries
You can use libraries from public pip packages and private wheel files. Libraries can be installed in many ways:
You can use libraries from public pip packages and private wheel files. Libraries can be installed in many ways:
- Inside a notebook: Only affects your session (most secure)
- On a shared cluster using requirement.txt file
- On private cluster (work in progerss)
- Inside a job cluster
A list of already installed packages for each runtime version is available in DataBricks documentation (use proper runtime version)
List already supported libraries
Run following command to see which libraries already installed in your cluster.
!pip list
Import new libraries to shared cluster
You can define required packages in the requirements.txt file and libraries will be installed on cluster level. The requirement.txt file is available in the /Workspace/Shared" folder. You can add any libraries you wish to be automatically installed when a cluster starts.
Define packages in the requirements.txt file, one library per line: Note: when updating libraries on cluster level, all users of that chared cluster is affected.
Example
python-dateutil==2.9.1
pandas==2.3.1
Installed packages are immediately available for use in your notebook session. Note: You are responsible for ensuring that only secure and trusted packages are included.
Install libraries on notebook level
You can install libraries that only applies to your notebook.
This example will downgrade your Pandas library from 2.3.1 running on cluster (because of your requirements.txt file) to 2.2.0 Example
%pip install pandas==2.2.0
%restart_python
import pandas as pd
print(pd.__version__)
Install private wheel
Git integration
Git integration is supported, allowing the requirements.txt file to be synced along with your notebooks.
If you're already managing a requirements.txt file via Git integration, you can submit a support ticket to update the path to point to your Git-managed file instead.