Devops/Git Integration

Databricks Repos provides built-in, repository-level integration with Git providers such as Azure DevOps. This enables you to develop code directly in Azure...

Databricks Repos provides built-in, repository-level integration with Git providers such as Azure DevOps. This enables you to develop code directly in Azure Databricks notebooks and synchronise your work with a remote Git repository.

Using Git is the recommended practice for collaborating on code in Databricks. It provides:

  • Versioning and change tracking.
  • Backup and recovery of source code.
  • A secure, controlled way of maintaining and sharing code.

Git also enables safe collaboration using branches and pull requests. Avoid using shared notebooks or folders for collaboration. Instead, use Git as the standard approach for team development.

Tip: Use the same authentication method for Azure DevOps and Databricks to ensure a seamless workflow.

Create and connect a Git repository

Follow these steps to connect your Azure DevOps repository to Databricks:

  1. Open your Azure DevOps Git repository.
  2. Navigate to the Git project you want to connect.
  3. Copy the HTTPS clone URL (for example, https://dev.azure.com/...).
  4. In Databricks, go to Workspace.
  5. In the left sidebar, expand Workspace and select Repos.
  6. Select Create Git Folder.
  7. In the dialog:
    • enter the Git repository URL
    • select Azure DevOps as the Git provider
    • enter the Git folder name For consistency, use the same name as your repository
    • optionally enable Sparse checkout mode if needed
  8. Create the Git folder.

Databricks will clone the repository into your workspace.

To complete authentication, follow the official documentation.

Where your Git folder is located

After connecting a repository, it appears in your workspace:

Workspace > Repos > Users > your username > repository name

This is your personal Git folder.

  • Perform all Git operations (branching, committing, switching branches) here.
  • Each user works in their own Git folder.
  • Do not perform Git operations in shared folders.

Recommended Git workflow in Databricks

Use a branch-based workflow when working with Git.

Work in your own Git folder

Always work in:

Workspace > Repos > Users > your username > repository

Do not collaborate by editing the same notebook or folder.

Create a branch

Always create a new branch for your work:

  1. Open your Git folder.
  2. Select the current branch (for example, main).
  3. Create a new branch (for example, feature/add-validation).

Use one branch per task.

Edit notebooks and scripts

Within your branch, you can:

  • edit notebooks
  • edit Python or SQL files
  • add new files

When creating notebooks:

  • use descriptive names
  • avoid default date-based names

Run and validate your code directly in Databricks.

Commit and push changes

After making changes:

  1. Open the Git panel in Databricks.
  2. Review modified files.
  3. Add a commit message.
  4. Commit and push your branch.

Keep commits small and focused.

The Git panel is visible only when you open a file inside a Git folder. You can access it from the top bar by selecting the branch name or file history.

Create a pull request

Do not merge directly in Databricks.

Instead:

  1. Open Azure DevOps.
  2. Create a pull request to main.
  3. Request review and merge.

Update your workspace

After your changes are merged:

  1. Switch to main
  2. Pull the latest changes

Always commit your work before pulling.

Working locally and in Databricks with Git

Depending on your workflow, you may work either locally (for example, in VS Code) or directly in Databricks.

Key differences

When running code locally vs in Databricks:

  • file access paths differ

    • local: ./data/file.csv
    • Databricks: /dbfs/... or abfss://...

  • Spark availability differs

    • Databricks provides a managed Spark cluster.
    • local environments require Databricks Connect for Spark access.

  • environment configuration differs

    • Local Python version and libraries depend on your machine.
    • Databricks uses cluster runtime versions.

Working locally (VS Code)

Use local development for:

  • writing modular Python or SQL code
  • refactoring and structuring projects
  • unit testing

Typical workflow:

  1. Clone the repository locally.
  2. Open it in VS Code.
  3. Create a branch: git checkout -b <your-branch-name>.
  4. Develop and test code locally.

Recommended: use Databricks Connect to run code on a Databricks cluster from your local environment.

Working in Databricks (Repos)

Use Databricks Repos for:

  • data exploration
  • ad-hoc analysis
  • notebook-based development
  • testing code close to data

Typical workflow:

  • open your Git folder
  • switch to the correct branch
  • edit notebooks or files
  • commit and push changes

Practical guidance

  • Handle file paths explicitly depending on environment.
  • Use Spark or SQL to filter data before processing in Pandas.

Conceptual model: code, data, and environment

Understanding how code and data are organised in the Data Platform is essential for correct workflows.

Code

Store code in:

  • Git repositories (primary and durable location).
  • Workspace files (notebooks, SQL, Python scripts).

Git should always be the source of truth for code.

Data

Store data in:

  • Unity Catalog tables (datasets)
  • volumes (File storage)

Do not store data in Git repositories.

Temporary or experimental data

  • Small test data may exist in the workspace.
  • Do not rely on workspace storage for persistence.

Important considerations

  • Workspace files are not a durable storage mechanism.
  • Deleting a workspace removes workspace files.
  • Git ensures persistence and version control of code.

Conceptual model: Data Workbench and Databricks

In the Veracity Data Platform, Data Workbench data is directly available in Databricks through Unity Catalog.

Mapping

  • Data Workbench datasets → Databricks tables.
  • Available under default.Tables.
  • Data Workbench files and folders → Databricks volumes.
  • Available under default.Volume.

These are the same underlying data, not copies.

What this enables

Unity Catalog provides:

  • governance
  • lineage tracking
  • auditing
  • fine-grained access control

Important behaviour

  • Updating an existing table in Databricks updates the corresponding Data Workbench dataset.
  • New tables must be written correctly to become Data Workbench datasets.

Best practices

  • Always use Git for collaboration.
  • Use one Git folder per user.
  • Do not work directly on main.
  • Use one branch per task.
  • Use pull requests for merging changes.
  • Do not co-edit notebooks or folders.
  • Keep commits small and focused.
  • Use Databricks for data exploration.
  • Use local environments for structured development.
  • Keep code in Git, not in workspace only.
  • Minimise dependencies and keep environments consistent.

More information

For more information about best practices for Databricks Repos, see best practices.