Connectivity to other datasources

Migrate into data lake

The recommended approach is to ingest data to the Veracity data lake using APIs or SAS keys. These data are then available directly for Analytics:

• Datasets are visible as tables in Databricks Catalog.
• File storage is visible as Volume/File storage in Databricks Catalog.

Share from another workspace

If the data you want to access from Analytics is not in the related Data Workbench workspace, but in another workspace in Veracity Data Platform, sharing is recommended. Set up a workspace-level share from one Veracity workspace to another (B2B sharing). If Workspace B has Analytics enabled, data shared from Workspace A to Workspace B is available in Databricks Catalog. Note: Currently this applies to datasets only. Shared files from File storage need to be accessed using SAS keys.

Shared datasets are available in the Analytics Environment as a Table. Shared files will soon be available.

For more details about sharing of datasets, go here.

Connect using SAS keys

Data from other storage containers can be accessed from Analytics using a SAS key. See how to generate a SAS key from the portal. SAS keys can also be generated using the API endpoint.

Access single file using SAS key

from urllib.parse import urlparse, parse_qs
#the complete sas key from Data Workbench - should be stored as secret
dfs_url = "https://prdstorageconst01weu.dfs.core.windows.net/......"

parsed = urlparse(dfs_url)
sas_token = parsed.query  # everything after '?'
storage_account = parsed.hostname.split('.')[0] 
container = parsed.path.split('/')[1]  
file_path = '/'.join(parsed.path.split('/')[2:]) 

print(sas_token)
print(storage_account)
print(container)
print(file_path)

spark.conf.set(
    f"fs.azure.sas.{container}.{storage_account}.blob.core.windows.net",
    sas_token
)

# Build abfss:// path
abfss_path = f"wasbs://{container}@{storage_account}.blob.core.windows.net/{file_path}"
 
# Load CSV file with Spark
df_spark = spark.read.csv(abfss_path, header=True, inferSchema=True)
 
# Display the Spark DataFrame
df_spark.display()

Access folder using SAS key

from urllib.parse import urlparse, parse_qs
from pyspark.dbutils import DBUtils

#the complete sas key from Data Workbench - should be stored as secret
dfs_url = "https://prdstorageconst01weu.dfs.core.windows.net/......"

parsed = urlparse(dfs_url)
sas_token = parsed.query  # everything after '?'
storage_account = parsed.hostname.split('.')[0]  
container = parsed.path.split('/')[1]  
folder_path = '/'.join(parsed.path.split('/')[2:])  

print(sas_token)
print(storage_account)
print(container)
print(folder_path)

# Set Spark config for abfss
spark.conf.set(
    f"fs.azure.sas.{container}.{storage_account}.blob.core.windows.net",
    sas_token
)

dbutils = DBUtils(spark)

files = dbutils.fs.ls(f"wasbs://{container}@{storage_account}.blob.core.windows.net/{folder_path}/")
for f in files:
    print(f.path)
    # Load CSV file with Spark
    df = spark.read.csv(f.path, header=True, inferSchema=True) 
    # Display the Spark DataFrame
    df.display()

How to connect existing databases

The recommended approach is to migrate these databases to a managed data lake or SQL Warehouse in Veracity Data Platform.

JDBC (Java Database Connectivity) and ODBC (Open Database Connectivity) are standard driver/protocol interfaces for connecting to databases from applications and analytics tools.

JDBC and ODBC: Databricks provides built-in integrations to many cloud-native data systems, as well as extensible JDBC support to connect to other data systems.

The source system should be on Verit and Verit Networks to ensure connectivity.

If the source system is not on Verit, the data can be queried using REST APIs directly from the Analytics environment.

Connectors

When datasets are available in Veracity Data Platform through connectors, data is not ingested to the platform. For the Analytics environment to use these data, the data must be queried using the Veracity query APIs.