Compute configurations
Databricks compute is centered around clusters, which are groups of virtual machines provisioned on demand to execute data workloads. A cluster consists of a...
Databricks compute is centered around clusters, which are groups of virtual machines provisioned on demand to execute data workloads. A cluster consists of a driver node, which coordinates execution, and one or more worker nodes, which perform the actual data processing. This architecture enables distributed computation, allowing Databricks to efficiently handle workloads ranging from small, single-node data transformations to large-scale distributed processing and machine learning.
Databricks compute provides a flexible and scalable foundation that supports a wide range of workloads—from simple Python-based data processing to large-scale Spark jobs and GPU-accelerated machine learning—while balancing usability, performance, and cost management.
Veracity supports multiple Databricks tiers, each accompanied by a set of standardized compute policies. Compute policies ensure consistency, governance, and cost control. These policies restrict and guide how clusters are created, including which VM types are allowed, scaling limits, and runtime versions. This allows flexibility for advanced users while maintaining standardized and efficient resource usage across the platform.
- Standard: All users within a Databricks environment operate on a shared, autoscaling cluster. This cluster is provisioned by Data Workbench by default, and its Databricks Runtime is automatically updated to the latest Long-Term Support (LTS) version.
- Advanced: Enables policy-based cluster creation. Users can provision clusters within defined compute policies, allowing controlled flexibility. Users are responsible for selecting and maintaining their Databricks Runtime version. Policies provide a curated selection of appropriate VM types and enforce limits on cluster scaling.
Standard
This tier provides a single all-purpose compute cluster shared across users. The configuration is designed to balance frequent workloads (requiring a slightly more capable driver node) with support for larger-scale SQL and PySpark processing.
Key cluster configuration:
- Driver: D8asd v5 (8xAMD CPU; 32 GB RAM)
- Workers (Autoscaling 1 – 6): D4asd v5 (4xAMD CPU; 16 GB RAM)
- Databricks runtime: Always the latest LTS – automatically patched by the Data Workbench team
- Photon: yes
- Auto shutdown: 40 mins
The estimated hourly cost for this cluster ranges from $2.70 (1 active worker) to $9.45 (6 active workers).
If scheduled jobs are required, dedicated job clusters should be used (Advanced tier).
Advanced
The Standard cluster remains available in this tier. In addition, users can provision custom clusters within defined compute policies:
VMs available for selection: NOTE: Prices updated per 2026-06-18. For current prices, use the Azure calculator. Prices listed are for interactive clusters. Job clusters are somewhat cheaper.
Personal Compute
This option supports single-node clusters and is optimized for workloads where Spark is not required. Typical use cases include small datasets and processing primarily implemented in Python libraries such as Pandas or Polars.
DBU rate: $0.55/DBU — All-Purpose Compute Premium tier (azure.microsoft.com/en-us/pricing/details/databricks/). Azure VM prices are Linux pay-as-you-go, region: westeurope.
Prices updated: 2026-06-18. For current prices use the Azure Pricing Calculator.
Single-node clusters. Optimised for Python (Pandas/Polars) workloads without Spark.
| Node VM type | Description | # Nodes | DBU / node / h | Databricks USD / node / h | Azure USD / node / h | Total USD / node / h |
|---|---|---|---|---|---|---|
| D4ads v5 (no Photon) | Small data and simple Python scripts | 1 | 1.0 | 0.55 | 0.25 | 0.80 |
| D4ads v5 (Photon) | + advanced Spark features | 1 | 2.0 | 1.10 | 0.25 | 1.35 |
| D8ads v5 (no Photon) | Medium data and simple multi-threaded Python scripts | 1 | 2.0 | 1.10 | 0.50 | 1.60 |
| D8ads v5 (Photon) | + advanced Spark features | 1 | 4.0 | 2.20 | 0.50 | 2.70 |
| D16ads v5 (no Photon) | Medium data and large Pandas and Polars | 1 | 4.0 | 2.20 | 1.00 | 3.20 |
| D16ads v5 (Photon) | + advanced Spark features | 1 | 8.0 | 4.40 | 1.00 | 5.40 |
| D32ads v5 (no Photon) | Extremely large data (consider proper cluster) | 1 | 8.0 | 4.40 | 2.00 | 6.40 |
| D32ads v5 (Photon) | + advanced Spark features | 1 | 16.0 | 8.80 | 2.00 | 10.80 |
| E16ads v5 (no Photon) | Memory-intensive operations (Pandas, Polars) | 1 | 5.5 | 3.03 | 1.26 | 4.29 |
| E16ads v5 (Photon) | + advanced Spark features | 1 | 11.0 | 6.05 | 1.26 | 7.31 |
| E32ads v5 (no Photon) | Memory-intensive operations (consider proper cluster) | 1 | 11.0 | 6.05 | 2.53 | 8.58 |
| E32ads v5 (Photon) | + advanced Spark features | 1 | 22.0 | 12.10 | 2.53 | 14.63 |
NOTE: Photon is a vectorized execution engine built into Databricks that significantly improves performance for SQL and Spark workloads. It is optimized for modern hardware and can reduce execution time and overall cost for many data processing tasks. Photon is particularly beneficial for data engineering and SQL-heavy workloads, but may provide limited benefit for pure Python (non-Spark) processing.
Scalable Compute
Designed for large-scale data processing workloads primarily implemented in SQL and PySpark, typically involving datasets in the tens of gigabytes or larger. Cluster autoscaling is recommended to handle varying workload sizes efficiently.
In this configuration, the minimum cluster size is always two nodes (driver + worker).
DBU rate: $0.55/DBU — All-Purpose Compute Premium tier (azure.microsoft.com/en-us/pricing/details/databricks/). Azure VM prices are Linux pay-as-you-go, region: westeurope.
Prices updated: 2026-06-18. For current prices use the Azure Pricing Calculator.
Multi-node autoscaling clusters for SQL and PySpark workloads.
| Node VM type | Description | # Nodes | DBU / node / h | Databricks USD / node / h | Azure USD / node / h | Total USD / node / h |
|---|---|---|---|---|---|---|
| F4s (no Photon) | Very basic, low-memory Spark processing | 2-10 | 0.5 | 0.28 | 0.23 | 0.51 |
| F8s (no Photon) | Very basic, low-memory Spark processing | 2-10 | 1.0 | 0.55 | 0.45 | 1.00 |
| D4ads v5 (no Photon) | Medium data Spark processing | 2-10 | 1.0 | 0.55 | 0.25 | 0.80 |
| D4ads v5 (Photon) | + advanced Spark features | 2-10 | 2.0 | 1.10 | 0.25 | 1.35 |
| D8ads v5 (no Photon) | Big data Spark processing | 2-10 | 2.0 | 1.10 | 0.50 | 1.60 |
| D8ads v5 (Photon) | + advanced Spark features | 2-10 | 4.0 | 2.20 | 0.50 | 2.70 |
| E8ads v5 (no Photon) | Big data Spark processing (memory-intensive aggregations) | 2-10 | 5.5 | 3.03 | 0.63 | 3.66 |
| E8ads v5 (Photon) | + advanced Spark features | 2-10 | 5.5 | 3.03 | 0.63 | 3.66 |
ML Compute (GPU)
This compute category is intended exclusively for machine learning workloads that require GPU acceleration, such as model training on large datasets. Due to general GPU availability constraints, higher-end SKUs (e.g., A100 or H100) are not included. The policy for GPU clusters is intentionally more flexible, and users are expected to select resources based on their specific workload requirements.
DBU rate: $0.55/DBU — All-Purpose Compute Premium tier (azure.microsoft.com/en-us/pricing/details/databricks/). Azure VM prices are Linux pay-as-you-go, region: westeurope.
Prices updated: 2026-06-18. For current prices use the Azure Pricing Calculator.
GPU-accelerated clusters for machine learning workloads.
| Node VM type | Description | # Nodes | DBU / node / h | Databricks USD / node / h | Azure USD / node / h | Total USD / node / h |
|---|---|---|---|---|---|---|
| NC4as T4 v3 (no Photon) | Entry-level ML training / fine-tuning, dev notebooks, small CV/NLP models, light hyperparameter tuning | 1-2 | 1.0 | 0.55 | 0.66 | 1.21 |
| NC16as T4 v3 (no Photon) | Cost-effective training for medium models, larger batches, multi-worker distributed training | 2-4 | 2.5 | 1.38 | 1.50 | 2.88 |
| NV36ads_A10_v5 (no Photon) | Strong GPU + high CPU/RAM: faster training, larger models, larger batches, heavy preprocessing | 1-2 | 4.0 | 2.20 | 4.16 | 6.36 |
Python versions
Databricks validates and ships a specific Python version with each Runtime release.
Databricks Runtime 18 LTS uses Python 3.12 on the cluster. All notebooks, jobs, and Python code executed on cluster drivers and executors run under the Databricks-managed Python 3.12 environment.
As of Databricks Runtime 18 LTS, Apache Spark 4.1.0 is included. While Spark 4.1 introduces compatibility improvements for newer Python versions, including Python 3.14, this primarily applies to client-side tools such as Databricks Connect. Python 3.14 is not the runtime Python version used by Databricks clusters in Runtime 18.x.
(For runtime version details, see:)[https://docs.databricks.com/aws/en/release-notes/runtime/]
If your scripts require Python 3.14-specific functionality, they cannot currently run natively on a Databricks Runtime 18 cluster. Running them externally (for example via Databricks Connect) is the primary supported approach.
The options are:
Preferred: Make the code compatible with Python 3.12: If the scripts do not depend on Python 3.13/3.14-specific language features, they will likely run on Databricks Runtime 18 without modification.
Use Databricks Connect: Run the application from a local or external environment using Python 3.14.Submit Spark operations to the Databricks cluster through Databricks Connect.This is the most straightforward approach when Python 3.14 support is a hard requirement.