Filestorage

Volumes govern access to non-tabular data. This is where all files from Veracity File storage are listed. Note: See the exception for shared files and folders below. The same file storage is used from Data Workbench, so no action is needed to sync new files back to Data Workbench.

Shared files/folders

Shared files and folders from other workspaces currently do not appear in the Databricks Volume section. This feature depends on a Databricks update planned this year. The workaround for using shared files and folders in Analytics is to use SAS keys:

1. Share files or folders from one Data Workbench workspace to the workspace with read/write access.
2. Create a SAS token from the Data Workbench file or folder.
3. Access shared files in Databricks using the SAS token. See the code below.

Read files from Volume

The choice between PySpark and Pandas depends on the size and complexity of your dataset and the nature of your application. If you are working with small to medium-sized datasets, Pandas is a good choice. If you are dealing with big data or real-time processing, PySpark is a better option. Pandas loads data in memory before running queries, so it can only query datasets that fit into memory. Spark can query datasets that are larger than memory by streaming the data and incrementally running computations.

Note: openpyxl provides fine-grained control over reading and writing Excel files. The read_only mode significantly improves performance when reading large files. Reading CSV is faster than reading XLSX.

import pandas as pd
inputfile = "volume path - find file in Volume and get column path"
#If reading from widget
#inputfile = dbutils.widgets.get("inputFileName")

pDf = pd.read_excel(inputfile, sheet_name="Sheet1") 
display(pDf)

Synchronize files with Data workbench

There is no action required to synchronize files between Veracity Data Platform File storage and the Databricks environment. Files uploaded to Veracity Data Platform File storage are visible in Databricks under Data Catalog/Default/Volumes. New files stored in Volume in Databricks are visible in Data Platform File storage in the same subfolders.

Write files to Volume

If creating a new file in Volume, you can create a new directory from the workspace or from a notebook.

import os
os.mkdir('/Volumes/<path>/default/filestorage/MyDir')

##outputfilename is stored in widget
filename = dbutils.widgets.get("outputfilepath")
df.to_csv(filename, index= False)

Read shared folder from other workspace

The following example allows Databricks to process all files in a shared folder and subfolder. The folder needs to be shared with read/write access in order for the receiver to generate a SAS key in Data Workbench.

#the complete sas key from Data Workbench - should be stored as secret
dfs_url = "https://prdstorageconst01weu.dfs.core.windows.net/....."

from urllib.parse import urlparse
parsed = urlparse(dfs_url)
sas_token = parsed.query  # everything after '?'
storage_account = parsed.hostname.split('.')[0]  
container = parsed.path.split('/')[1]  
folder_path = '/'.join(parsed.path.split('/')[2:])  

def read_csv_recursive(path):
    items = dbutils.fs.ls(path)

    for item in items:
        full_path = item.path

        if item.size == -1:
            print(f"Folder: {full_path}")
            read_csv_recursive(full_path)

        else:
            # It's a file → check if it's CSV
            if full_path.lower().endswith(".csv"):
                print(f"Reading CSV file: {full_path}")
                df = spark.read.csv(full_path, header=True, inferSchema=True)
                display(df)
            else:
                print(f"Skipping non-CSV file: {full_path}")


# Set Spark config for abfss
spark.conf.set(
    f"fs.azure.sas.{container}.{storage_account}.blob.core.windows.net",
    sas_token
)
rootPath = f"wasbs://{container}@{storage_account}.blob.core.windows.net/{folder_path}/"
read_csv_recursive(rootPath)

Read shared file from other workspace

#the complete sas key from Data Workbench - should be stored as secret
dfs_url = "https://prdstorageconst01weu.dfs.core.windows.net/...."

from urllib.parse import urlparse
parsed = urlparse(dfs_url)
sas_token = parsed.query  # everything after '?'
storage_account = parsed.hostname.split('.')[0]  
container = parsed.path.split('/')[1]  
folder_path = '/'.join(parsed.path.split('/')[2:])  

# Set Spark config for abfss
spark.conf.set(
    f"fs.azure.sas.{container}.{storage_account}.blob.core.windows.net",
    sas_token
)
filepath = f"wasbs://{container}@{storage_account}.blob.core.windows.net/{folder_path}/"
item = dbutils.fs.ls(filepath)
df = spark.read.csv(item[0].path, header=True, inferSchema=True)
display(df)