Ingest files

This feature lets you upload files of any type into your Azure Data Lake. Files are stored as unstructured data in Veracity File storage and cannot be queried using SQL-like operations. Subscriptions to Veracity File Storage is required.

API endpoints

Note: The Ingest api-endpoints are different for uploading files to filestorage vs uploading datasets. To browse the api, go here. See api section File Storages.

All response formats and status codes are documented in the API Explorer

Base url: https://api.veracity.com/veracity/dw/gateway/api/v2

Authentication: Authentication and subscription key is required for each api request, see details.

When authenticating using service account, the service account needs WRITE permissions.

Ingest/Upload process

Using the apis these are the three steps to follow:

• Step 1: Get SAS token uri from Veracity api (Note: different endpoint than for getting SAS Uri for datasets).
• Step 2: Upload file from localpath to storage using received SAS token uri.
• Step 3: Update metadata (optional).

Response codes and response payloads

All response formats and status codes are documented in the API Explorer, where each endpoint is listed. You can browse the Upload files API there for detailed information, see section File Storages.

Step 1: Get SAS URI for Filestorage

Using the Veracity token from authentication, ensure user/service account has Write access.

Input payload Payload options:

• path is optional. It is the path to the resource for which you're generating the SAS token. If you don't provide a path, the default path will be used. The default path is the ContainerName (i.e. root level).
• readOrWritePermission : Set write to be able to Write. Ensure service account has Write access (i.e. has Admin role in workspace).
• StartsOn is optional. It is the start date when the SAS token becomes valid. If not provided, it takes the current UTC date time.
• ExpiresOn is when the SAS token stops being valid. It should be greated than StartsOn and can't be a past date.
• StorageName is optional. If used, it should be a valid name of a data set in File storage. If not provided, it takes the default internal storage data set.

Output Either full sas uri path or split into objects (resourceUri and sasToken) depending on whether '?format=object' is used in query string.

Python

import requests
import json
from datetime import datetime, timedelta
 
mySubscriptionKey = dbutils.secrets.get(scope="secrets", key="bkal_subKey")
workspaceId = "workspace id"
#if path not provided, root is used
dwbFolderName = "folder name" 
def get_sas_token(veracity_token, folder, workspace_id, subscription_key):
    base_url = "https://api.veracity.com/veracity/dw/gateway/api/v2"
    endpoint = f"/workspaces/{workspace_id}/storages/sas?format=object"
    url = base_url + endpoint
    expires_on = (datetime.utcnow() + timedelta(hours=2)).isoformat() + "Z"
 
    payload = {
      "path": folder,
      "readOrWritePermission": "Write",
      "expiresOn": expires_on
    }

    headers = {
        "Content-Type": "application/json",
        "Ocp-Apim-Subscription-Key": subscription_key,
        "Authorization": f"Bearer {veracity_token}",
        "User-Agent": "python-requests/2.31.0"
    }

    try:
        response = requests.post(url, json=payload, headers=headers)
        response.raise_for_status()
        return response.json()
    except requests.exceptions.RequestException as e:
        print(f"Error fetching SAS token: {e}")
        return None

sas_uri_object = get_sas_token(veracity_token=veracityToken, folder=dwbFolderName, workspace_id=workspaceId, subscription_key=mySubscriptionKey)
sasToken = sas_uri_object.get("sasToken")
resourceUri = sas_uri_object.get("resourceUri")

C#

using Azure.Storage.Files.DataLake;
using System.Net.Http.Headers;
using System.Text;
using Newtonsoft.Json;

public async Task<string> GetSASUri(string veracityToken, string folder, string workspaceId, string subscriptionKey)
{
    string url = $"https://api.veracity.com/veracity/dw/gateway/api/v2/workspaces/{workspaceId}/storages/sas";
    // use ?format=object if you want an object to be returned with parsed SAS URI

    DateTime expiresOn = DateTime.UtcNow.AddHours(1);
    string expiresOnStr = expiresOn.ToString("O");

    //path is folder name in storage container
    var postData = new Dictionary<string, string>
    {
        {"path", folder},
        {"readOrWritePermission", "Write"},
        {"expiresOn", expiresOnStr }
    };

    string jsonString = JsonConvert.SerializeObject(postData);
    HttpContent content = new StringContent(jsonString, Encoding.UTF8, "application/json");

    if (_httpClient == null)
        _httpClient = new HttpClient();

    _httpClient.BaseAddress = new Uri(url);

    _httpClient.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", subscriptionKey);
    _httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", veracityToken);
    try
    {
        var result = await _httpClient.PostAsync(url, content);
        if (result.IsSuccessStatusCode)
        {
            string sasKey = result.Content.ReadAsStringAsync().Result;
            return sasKey.Trim('"');
        }
    }
    catch (Exception ex)
    {

    }
    return null;
}

Step 2: Upload file

Upload file using SAS URI from Step 2.

File is uploaded using Microsoft libraries.

Python

from azure.storage.filedatalake import DataLakeFileClient
from urllib.parse import urlparse
import os
from urllib.parse import urlparse

localFilePath = "local-file-path"
#target file name does not need to be same as source path
targetFileName = "MyTestData.csv"

#sas token from step 2
sas_token = sasToken

# === PARSE resource uri from step 2 ===
parsed_url = urlparse(resourceUri)
# get account combining schema (https) and netlock
account_url = f"{parsed_url.scheme}://{parsed_url.netloc}"
file_system_name = parsed_url.path.strip("/").split("/")[0]
folder_path = "/".join(parsed_url.path.strip("/").split("/")[1:])

# === COMBINE FOLDER + FILE NAME ===
file_path = f"{folder_path}/{targetFileName}"

# === CREATE FILE CLIENT ===
file_client = DataLakeFileClient(
    account_url=account_url,
    file_system_name=file_system_name,
    file_path=file_path,
    credential=sas_token
)
# === CREATE FILE AND UPLOAD ===
file_client.create_file()

with open(localFilePath, "rb") as file_data:
    result = file_client.upload_data(file_data, overwrite=True)

# If an etag is returned after an upload, it means the file was successfully written to Azure Data Lake.
etag = result.get("etag")
if result.get("etag") and result.get("last_modified"):
    print(f"✅ Upload successful with etag {etag}.")
else:
    print(f"⚠️ Upload failed")

C#

public async Task<string> UploadFileToFileStorage(string sasToken, string filepath, Dictionary<string, string> metadata)
{
    var sasUri = new System.Uri(sasToken);
    var containerClient = new DataLakeDirectoryClient(sasUri);

    string remoteFileName = Path.GetFileName(filepath);
    var fileClient = containerClient.GetFileClient(remoteFileName);

    string repsonse_status = "Unknown";
    using (FileStream fsSource = new FileStream(filepath, FileMode.Open, FileAccess.Read))
    {
        bool overwrite = true;
        var response = await fileClient.UploadAsync(fsSource, overwrite, CancellationToken.None);
        repsonse_status = response.GetRawResponse().Status.ToString();
    }
    //metadata is optional
    if (metadata != null && metadata.Count > 0)
        await fileClient.SetMetadataAsync(metadata);

    return repsonse_status;
}

Step 3: Ingest metadata to existing file or folder

Use sas-token uri from Step 1 and use DataLakeFileClient to update metadata.

Python

from azure.storage.filedatalake import DataLakeFileClient
import os
from urllib.parse import urlparse

#If URI points to folder
targetFileName = "<filename in target system>"

#sas token and resource uri from step 2
sas_token = sasToken
resource_uri = resourceUri

# === PARSE resource uri from step 2 ===
parsed_url = urlparse(resource_uri)
# get account combining schema (https) and netlock
account_url = f"{parsed_url.scheme}://{parsed_url.netloc}"
file_system_name = parsed_url.path.strip("/").split("/")[0]
folder_path = "/".join(parsed_url.path.strip("/").split("/")[1:])

# === COMBINE FOLDER + FILE NAME ===
file_path = f"{folder_path}/{targetFileName}"

# === CREATE FILE CLIENT ===
file_client = DataLakeFileClient(
    account_url=account_url,
    file_system_name=file_system_name,
    file_path=file_path,
    credential=sas_token
)

#Example of metadata - is optional
metadata = {
    "Customer": "123",
    "Site": "ABC"    
}
result = file_client.set_metadata(metadata)
# If an etag is returned after an upload, it means the file was successfully written to Azure Data Lake.
etag = result.get("etag")
if result.get("etag") and result.get("last_modified"):
    print(f"✅ Upload metadata successful with etag {etag}.")
else:
    print(f"⚠️ Upload metadata failed")

C#

    async Task UploadMetadataToFile(string sasToken, string remoteFileName,  Dictionary<string, string> metadata)
    {
        var sasUri = new System.Uri(sasToken);
        var containerClient = new DataLakeDirectoryClient(sasUri);
                
        var fileClient = containerClient.GetFileClient(remoteFileName);
           
        if (metadata != null && metadata.Count > 0)
        {
            var response = await fileClient.SetMetadataAsync(metadata);
        }
    }