Ingest structured data
Datasets are structured data and the structure (columns) are defined using schemas. Data in datasets can be queried using query language.The following apis are...
Datasets are structured data and the structure (columns) are defined using schemas. Data in datasets can be queried using query language. The following apis are awailable for uploading structured dataset.
| Api | Description |
|---|---|
| Json ingest | Content of dataset uploaded as json structure in payload. For small to medium sized dataset |
| CSV upload | Direct upload of csv file using api-endpoint. For files < 50MB |
| SAS uri | For large files, request SAS uri using api and use sas uri to upload file directly to datalake using Azure storage apis optimized for large-scale transfers |
API endpoints
Note: The Ingest api-endpoints are different for uploading datasets vs uploading files to filestorage. To browse the api. See section Datasets Ingest.
Base url: https://api.veracity.com/veracity/dw/gateway/api/v2
Authentication: Authentication and subscription key is required for each api request, see details.
To authenticate and authorize your calls, you need API key and a bearer token here. To write data, the user/service account needs WRITE permissions (admin role). When creating a service account, its role is by default READER. To give it Write access, send request to Veracity support requesting Admin role to "service account id" in "workspace id".
Upload new dataset using Json
This is a synchronous call used for uploading a new dataset.
POST /workspaces/{workspace_id}/ingest/dataset/json
Payload example In this example the schema coontains columns dateTime, Temp, Press and Wind. The dataset is taggee with cusomer id 123
{
"datasetName": "Test",
"datasetDescription": "Test",
"schemaVersionId": "a5b49521-6067-467d-b252-8d1b6205a834",
"data": [
{
"row": {"DateTime": "2026-05-03T12:00:01", "Temp": 12, "Press": 232, "Wind": 3}
},
{
"row": {"DateTime": "2026-05-01T12:00:01", "Temp": 17, "Press": 2, "Wind": 21}
},
{
"row": {"DateTime": "2026-05-06T12:00:01", "Temp": 7, "Press": 244, "Wind": 1}
}
],
"tags": [
{
"key": "customer",
"values": [
"123"
]
}
]
}
Update existing dataset using Json
Operation supported: Update, Delete, Insert To be able to delete or update, the schema needs to have a key defined to indetify row uniquness.
- Operation Upsert: Will update rows with matching key, and will append new rows. When changing a row, all columns for the row must be provided.
- Operation Delete: Will delete rows with matching key.
- Operation Append: Will add rows to dataset where key is not defined.
PUT /workspaces/{workspace_id}/ingest/dataset/{datasetId}/json
Support multiple rows.
{
"data": [
{
"row": {
"column1": "value1",
"column2": "value2"
},
"operation": "Update"
}
]
}
Upload new dataset using CSV direct upload
Uploads the CSV file and metadata in one multipart request. The ingest runs synchronously and returns the id of the created dataset.
POST /workspaces/{workspace_id}/ingest/dataset/csv
Payload:
"datasetName": "My dataset",
"datasetDescription": "Optional description",
"schemaVersionId": "00000000-0000-0000-0000-000000000000",
"file": "data.csv",
"tags": [
{
"key": "env",
"values": [
"test"
]
}
]
}
C# example In this example a CSV file is uploaded and dataset is tagged with cusomer id 123
Note: The filename provided in payload "files" (i.e Test.csv) must be the same as the uploaded file. Currently "files" section only support one file.
if (!File.Exists(localFilePath))
throw new FileNotFoundException($"Local file '{localFilePath}' does not exist.");
string remoteFileName = Path.GetFileName(localFilePath);
var payload = new Dictionary<string, object?>
{
["datasetName"] = datasetName,
["datasetDescription"] = description,
["schemaVersionId"] = schemaVersionId,
["file"] = remoteFileName}
};
if (tags != null && tags.Count > 0)
payload["tags"] = tags.Select(kvp => new { key = kvp.Key, values = new[] { kvp.Value } }).ToArray();
using var form = new MultipartFormDataContent();
// The JSON job metadata must be the first part and named "request".
var payloadJson = JsonSerializer.Serialize(payload, CamelCase);
form.Add(new StringContent(payloadJson, Encoding.UTF8, "application/json"), "request");
// CSV content as a file part; the file name must match files[].fileName above.
var fileContent = new ByteArrayContent(await File.ReadAllBytesAsync(localFilePath));
fileContent.Headers.ContentType = new MediaTypeHeaderValue("text/csv");
form.Add(fileContent, "file", remoteFileName);
var request = new HttpRequestMessage(HttpMethod.Post,
config.BaseUrl.TrimEnd('/') + $"/workspaces/{workspaceId}/ingest/dataset/csv")
{
Content = form
};
request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", token);
if (!string.IsNullOrWhiteSpace(config.ApiKey))
request.Headers.Add("Ocp-Apim-Subscription-Key", config.ApiKey);
//using var response = await httpClient.SendAsync(request);
using var response = httpClient.Send(request);
var content = response.Content.ReadAsStringAsync().GetAwaiter().GetResult();
if (!response.IsSuccessStatusCode)
throw new ApiException(response.StatusCode, content);
string datasetId = "";
using (var doc = JsonDocument.Parse(content))
{
var root = doc.RootElement;
if (root.TryGetProperty("id", out var id) && id.ValueKind == JsonValueKind.String)
datasetId = id.GetString() ?? "";
else if (root.TryGetProperty("datasetId", out var dsId) && dsId.ValueKind == JsonValueKind.String)
datasetId = dsId.GetString() ?? "";
}
return datasetId;
In the above example tags input is Dictionary of key/values.
Update existing dataset using CSV direct upload
Uploads the CSV file and metadata in one multipart request. The uploded file contains the changes to be done to the dataset. When updating a dataset, you can use one of the following operations:
- Upsert: Update existing rows in dataset and ingest new rows (require that row keys are defined in schema).
- Delete: Delete rows in dataset where row keys match with row keys in file uploaded (require that row keys are defined in schema).
- Overwrite: Owerwrite existing dataset with new content from file uploaded.
- Append: add data to dataset with content from file uploaded. This operation can only be used on schemas that do NOT have schema keys.
PUT /workspaces/{workspace_id}/ingest/dataset/{datasetId}/csv
Same code example as above, but set correct operation. Currently method supports only single file upload.
{
"files": [
{
"fileName": "data.csv",
"operation": "Append"
}
],
}
Upload new dataset using SAS URI
This is a two step approach where you first register teh dataset and receives a upload URI to be used to upload the CSV. Tis method should be used for files > 100 MB. Using the apis these are the steps to follow:
- Authenticate towards Veracity api using service account (service principle). See code example.
- Step 1: Define dataset with metadata and get SAS token uri from Veracity api (Note: different endpoint than for getting SAS Uri for filestorage).
- Step 2: Read CSV file from your location and upload file to storage using received SAS token uri.
- Step 3: Start job (Note: only if you set startAutomatically = false in step 2).
- Step 4: Verify status on upload (optional).
When uploading a dataset it is the schema version id that is used (and not schema id). When using the api, schema version id is received in response. However, when schema is created in Data Workbench portal, you only see the schema_id (in url). To get schema version id, use schema Id and query the GET endpoint.
Step 1: Prepare dataset for upload
Define dataset with metadata and get SAS URI for uploading csv file Using the Veracity token from step 1, ensure user/service account has Write access. This step prepare upload operation and get SAS URI to upload file content. Ensure you have schema version id, see how to get it
Payload
- datasetName: Name of dataset.
- SchemaVersionId: The id of the version of the schema.
- Files: Currently only support for one file.
- Operations: "Create" (when this is a new dataset).
- Filename used does not need to match the source filename (local filepath used in step 3), but must match the target filename used in step 3. I.e you can use a generic filename such as ImportData.csv
- DatasetDescription: Description field is optional.
- Tags: Tags are metadata for the dataset, and is an array of key/values. Tags are optional. In the example below; customer and site are keys and "ABC" and "123" are values.
- Start Automatically: If True, the job of processing the file uploaded (step 3) will start automatically. However, if set to False, you need to start the job with api-endpoint. This is will trigger the job faster.
- Provider: Use "BYOD" (will be made optional).
Output Job id and sas_uri job_id is used to get status of the upload
Python
import requests
import json
#from authentication
token = veracityToken
apiKey = "<api key>"
workspaceId = "<workspace id>"
schemaVersionId = "<schema version id>"
base_url = "https://api.veracity.com/veracity/dw/gateway/api/v2"
endpoint = f"/workspaces/{workspaceId}/ingest/dataset"
url = base_url + endpoint
#Name to see in DWb
datasetName = "WindData"
targetFileName = "ImportData.csv"
description = "some dataset description"
headers = {
"Content-Type": "application/json",
"Ocp-Apim-Subscription-Key": apiKey,
"Authorization": f"Bearer {token}",
"User-Agent": "python-requests/2.31.0"
}
payload = {
"datasetName": datasetName,
"datasetDescription": description,
"schemaVersionId": schemaVersionId,
"files": [
{
"fileName": targetFileName,
"operation": "Create"
}
],
"tags": [{
"Key": "Customer",
"Values": ["ABC"]
},
{
"Key": "Site",
"Values": ["123"]
}
],
"provider": "BYOD",
"startAutomatically": True
}
try:
response = requests.post(url, json=payload, headers=headers)
response.raise_for_status()
except requests.exceptions.RequestException as e:
print(f"Error fetching sas uri: {e}")
result = response.json()
# Extract the SAS token
sas_uri = result.get('sasToken')
job_id = result.get('jobId')
C#
using Azure.Storage.Files.DataLake;
using Newtonsoft.Json;
using System.Net.Http.Headers;
using System.Text;
using System.Text.Json;
//output is a tuple with sasTokenUri and jobId
public async Task<(string, string)> GetSASUriForCreateOperation(string datasetName, string description,
string schemaVersionId, string targetFileName, string workspaceId,
string veracityToken, string subscriptionKey)
{
string sasTokenUri = string.Empty;
string job_Id = string.Empty;
string baseUrl = "https://api.veracity.com/veracity/dw/gateway/api/v2";
string url = $"{baseUrl}/workspaces/{workspaceId}/ingest/dataset";
var token = veracityToken;
var fileOperations = new List<Dictionary<string, object>>
{
new Dictionary<string, object>{
["fileName"] = targetFileName,
["operation"] = "Create"
}
};
//Example of how metadata can be created on dataset
var tags = new List<Dictionary<string, object>> {
new Dictionary<string, object> {
["Key"] = "Customer",
["Values"] = new List<string> { "ABC" }
},
new Dictionary<string, object> {
["Key"] = "Site",
["Values"] = new List<string> { "123" }
}
};
//if start autoamtically = false, job must be started with seperate endpoint after fileupload
var payload = new Dictionary<string, object>
{
{"datasetName", datasetName},
{"datasetDescription", description},
{"schemaVersionId", schemaVersionId},
{"provider", "BYOD"},
{"startAutomatically", true },
{"files", fileOperations },
{"tags", tags }
};
string jsonString = JsonConvert.SerializeObject(payload);
HttpContent content = new StringContent(jsonString, Encoding.UTF8, "application/json");
if (_httpClient == null)
_httpClient = new HttpClient();
_httpClient.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", subscriptionKey);
_httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
try
{
var result = await _httpClient.PostAsync(url, content);
if (result.IsSuccessStatusCode)
{
string json = result.Content.ReadAsStringAsync().Result;
using JsonDocument doc = JsonDocument.Parse(json);
JsonElement root = doc.RootElement;
JsonElement jobId = root.GetProperty("jobId");
JsonElement sasUri = root.GetProperty("sasToken");
sasTokenUri = sasUri.ToString().Trim('"');
job_Id = jobId.ToString().Trim('"');
}
}
catch (Exception ex)
{
throw ex;
}
return (sasTokenUri, job_Id);
}
Step 2: Upload CSV file
Upload file using SAS URI from Step 2 We are using Microsoft libraries to upload data.
After file is uploaded and processing is triggered, use job_id from step 2 to request status of the processing.
- localFilepath is path for file to be uploaded
- targetFileName must be same as in step 2
Python
from azure.storage.filedatalake import DataLakeFileClient, DataLakeDirectoryClient
from urllib.parse import urlparse
import os
import asyncio
from azure.core.exceptions import ResourceExistsError
import uuid
from urllib.parse import urlparse
#Source file
localFilePath = "<local file path>"
#Same file name as from step 2
target_file_name = targetFileName
# sas uri from step 2
sas_folder_url = sas_uri
# Parse the SAS URI from step 2
parsed_url = urlparse(sas_folder_url)
account_url = f"{parsed_url.scheme}://{parsed_url.netloc}"
file_system_name = parsed_url.path.strip("/").split("/")[0]
folder_path = "/".join(parsed_url.path.strip("/").split("/")[1:])
sas_token = parsed_url.query
# === COMBINE FOLDER + FILE NAME ===
file_path = f"{folder_path}/{target_file_name}"
# === CREATE FILE CLIENT ===
file_client = DataLakeFileClient(
account_url=account_url,
file_system_name=file_system_name,
file_path=file_path,
credential=sas_token
)
# Create a new file (overwrite if exists)
file_client.create_file()
# Upload file content
try:
with open(localFilePath, 'rb') as file_data:
file_contents = file_data.read()
file_client.append_data(data=file_contents, offset=0, length=len(file_contents))
file_client.flush_data(len(file_contents))
print("✅ File uploaded successfully, check status for file processing using jobId from step 2")
except Exception as e:
print(f"❌ General error occurred: {e}")
C#
using Azure.Storage.Files.DataLake;
using Newtonsoft.Json;
using System.Net.Http.Headers;
using System.Text;
using System.Text.Json;
public async Task<int> UploadFileToDataset(string sasUriToken, string localFilepath, string targetFileName)
{
var sasUri = new System.Uri(sasUriToken);
string remoteFileName = targetFileName;
var containerClient = new DataLakeDirectoryClient(sasUri);
var containerFileClient = containerClient.GetFileClient(remoteFileName);
using (FileStream fsSource = new FileStream(localFilepath, FileMode.Open, FileAccess.Read))
{
var response = await containerFileClient.UploadAsync(fsSource, true, CancellationToken.None);
return response.GetRawResponse().Status;
};
}
Step 3: Start job processing
Start processing the uploaded file(s) and store as delta tables This step only applies when "StartAutomatically" = False in step 1. If you try to start a job which is scheduled to start automatically, request will fail. After job is started, use job_id to request status of the processing.
Use job_id from step 2
Python
token = veracityToken
base_url = "https://api.veracity.com/veracity/dw/gateway/api/v2"
endpoint = f"/workspaces/{workspaceId}/ingest/job/{job_id}/start"
url = base_url + endpoint
headers = {
"Content-Type": "application/json",
"Ocp-Apim-Subscription-Key": apiKey,
"Authorization": f"Bearer {token}",
"User-Agent": "python-requests/2.31.0"
}
try:
response = requests.post(url, json=None, headers=headers)
response.raise_for_status()
except requests.exceptions.RequestException as e:
print(f"Error starting job: {e}")
result = response.json()
C#
public async Task<string> StartProcessingJob(string jobId, string workspaceId, string veracityToken, string subscriptionKey)
{
string url = $"https://api.veracity.com/veracity/dw/gateway/api/v2/workspaces/{workspaceId}/ingest/job/{jobId}/start";
var token = veracityToken;
if (_httpClient == null)
_httpClient = new HttpClient();
_httpClient.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", subscriptionKey);
_httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
HttpContent content = null;
try
{
var result = await _httpClient.PostAsync(url, content);
if (result.IsSuccessStatusCode)
{
string status = result.Content.ReadAsStringAsync().Result;
return status;
}
}
catch (Exception ex)
{
}
return $"Cannot start job {jobId}";
}
Step 4: Check status
Since upload dataset takes 30-40 sec and processing is an asynchronous operation, status can be queried using status endpoint with the job_id from step 2.
Before upload is completed status will show: {'jobId': '<job_id>', 'status': 'Running', 'operations': ['Create'], 'notebookErrorMessage': '', 'validationError': []}
After completion status shows: {'jobId': 'job_id', 'status': 'Completed', 'operations': ['Create'], 'notebookErrorMessage': '', 'dataSetName': 'dataset name', 'datasetId': 'dataset_id', 'validationError': []}
If status is Failed, the response provides more descriptive failure information. Inspect:
errorMessagefor the overall failure summary.notebookErrorMessagefor details reported by the processing notebook.validationErrorfor file-validation results. Each item can includefileName,isValid, anderrorReason.
For example:
{
"jobId": "<job_id>",
"status": "Failed",
"operations": ["Create"],
"errorMessage": "File(s) failed to validate.",
"notebookErrorMessage": "",
"validationError": [
{
"fileName": "data.csv",
"isValid": false,
"errorReason": "<validation error details>"
}
]
}
Python
import requests
apiKey = "<api key>"
workspaceId = "<workspace id>"
base_url = "https://api.veracity.com/veracity/dw/gateway/api/v2"
endpoint = f"/workspaces/{workspaceId}/ingest/job/{job_id}/status"
url = base_url + endpoint
headers = {
"Content-Type": "application/json",
"Ocp-Apim-Subscription-Key": apiKey,
"Authorization": f"Bearer {veracityToken}",
"User-Agent": "python-requests/2.31.0"
}
try:
response = requests.get(url, headers=headers)
response.raise_for_status()
except requests.exceptions.RequestException as e:
print(f"Error fetching job status: {e}")
result = response.json()
C#
public async Task<string> GetJobStatus(string jobId, string workspaceId, string veracityToken, string subscriptionKey)
{
string url = $"https://api.veracity.com/veracity/dw/gateway/api/v2/workspaces/{workspaceId}/ingest/job/{jobId}/status";
var token = veracityToken;
if (_httpClient == null)
_httpClient = new HttpClient();
_httpClient.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", subscriptionKey);
_httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
try
{
var result = await _httpClient.GetAsync(url);
if (result.IsSuccessStatusCode)
{
string status = result.Content.ReadAsStringAsync().Result;
return status;
}
}
catch (Exception ex)
{
}
return $"Cannot retrieve status for job {jobId.ToString()}";
}
Complete upload example python
import os
import time
import requests
import json
from dotenv import load_dotenv
load_dotenv()
token_url = "https://login.microsoftonline.com/dnvglb2cprod.onmicrosoft.com/oauth2/token"
client_id = os.getenv("CLIENT_ID")
client_secret = os.getenv("CLIENT_SECRET")
apiKey = os.getenv("API_KEY")
workspaceId = os.getenv("WORKSPACE_ID")
schemaId = os.getenv("SCHEMA_ID")
base_url = "https://api.veracity.com/veracity/dw/gateway/api/v2"
# Token payload for authentication request
token_payload = {
"grant_type": "client_credentials",
"client_id": client_id,
"client_secret": client_secret,
"resource": "https://dnvglb2cprod.onmicrosoft.com/83054ebf-1d7b-43f5-82ad-b2bde84d7b75"
}
# Function to retrieve access token
def get_token():
try:
response = requests.post(token_url, data=token_payload, verify=False)
return response.json()["access_token"]
except Exception as e:
print(f"Failed to retrieve access token: {e}")
token = get_token()
headers = {
"Content-Type": "application/json",
"Ocp-Apim-Subscription-Key": apiKey,
'Authorization': f"Bearer {token}",
"User-Agent": "python-requests/2.31.0"
}
def get_schema_version_id(workspace_id, headers):
url = f"{base_url}/workspaces/{workspace_id}/schemas/{schemaId}"
try:
response = requests.get(url, headers=headers, verify=False)
response.raise_for_status()
result = response.json()
# Get the list of schema versions
schema_versions = result.get("schemaVersions", [])
# find version with isDefault == True
default_version = next(
(sv for sv in schema_versions if sv.get("isDefault") is True),
None
)
if default_version:
return default_version.get("id")
else:
print("No default schema version found.")
return None
except requests.exceptions.RequestException as e:
print(f"Error fetching schema version ID: {e}")
return None
#Name to see in DWb
datasetName = "<dataset name>"
targetFileName = "name of file" #not path
description = "<some description >"
payload = {
"datasetName": datasetName,
"datasetDescription": description,
"files": [
{
"fileName": targetFileName,
"operation": "Create"
}
],
"startAutomatically": True,
"provider": "BYOD"
}
# ─────────────────────────────────────────────────────────────────────────────
# Main
# ─────────────────────────────────────────────────────────────────────────────
try:
# Find active version id for the schema
schemaVersionId = get_schema_version_id(workspaceId, headers)
payload["schemaVersionId"] = schemaVersionId
# prepare dataset ingest by requesting a sas uri for the file upload
endpoint = f"/workspaces/{workspaceId}/ingest/dataset"
url = base_url + endpoint
response = requests.post(url, json=payload, headers=headers, verify=False)
print(f"Status: {response.status_code}")
response.raise_for_status()
except requests.exceptions.RequestException as e:
print(f"Error fetching sas uri: {e}")
raise
result = response.json()
# Extract the SAS token and job id
sas_uri = result.get('sasToken')
job_id = result.get('jobId')
## Upload local file to the provided SAS URI
from azure.storage.filedatalake import DataLakeFileClient, DataLakeDirectoryClient
from urllib.parse import urlparse
from azure.core.exceptions import ResourceExistsError
from urllib.parse import urlparse
#Source file
localFilePath = "<local path for file to be uploaded"
target_file_name = targetFileName
parsed_url = urlparse( sas_uri)
account_url = f"{parsed_url.scheme}://{parsed_url.netloc}"
file_system_name = parsed_url.path.strip("/").split("/")[0]
folder_path = "/".join(parsed_url.path.strip("/").split("/")[1:])
sas_token = parsed_url.query
# === COMBINE FOLDER + FILE NAME ===
file_path = f"{folder_path}/{target_file_name}"
# === CREATE FILE CLIENT ===
file_client = DataLakeFileClient(
account_url=account_url,
file_system_name=file_system_name,
file_path=file_path,
credential=sas_token
)
# Create a new file (overwrite if exists)
file_client.create_file()
# Upload file content
try:
with open(localFilePath, 'rb') as file_data:
file_contents = file_data.read()
file_client.append_data(data=file_contents, offset=0, length=len(file_contents))
file_client.flush_data(len(file_contents))
print("✅ File uploaded successfully, check status for file processing using jobId from step 2")
except Exception as e:
print(f"❌ General error occurred: {e}")
while True:
url = f"{base_url}/workspaces/{workspaceId}/ingest/job/{job_id}/status"
try:
response = requests.get(url, headers=headers, verify=False)
response.raise_for_status()
result = response.json()
status = result.get("status", "")
if status.lower() == "completed":
print(" ✓ Ingest job completed!")
break
if status.lower() == "failed":
print(" ❌ Ingest job failed!")
break
print(f" Status: {status} — retrying in 20 seconds …")
time.sleep(20)
except Exception as e:
print(f"❌ General error occurred: {e}")
break
Update existing datasets using SAS URI
When updating a dataset, you can use one of the following operations:
- Upsert: Update existing rows in dataset and ingest new rows (require that row keys are defined in schema).
- Delete: Delete rows in dataset where row keys match with row keys in file uploaded (require that row keys are defined in schema).
- Overwrite: Owerwrite existing dataset with new content from file uploaded.
- Append: add data to dataset with content from file uploaded. This operation can only be used on schemas that do NOT have schema keys.
Using the apis these are the steps to follow:
- Authenticate towards Veracity api using service account (service principle)
- Step 1: Prepare update operation on dataset with metadata,define if operation is Upsert, delete, overwrite or append, and get SAS token uri
- Step 2: Read CSV file from your location and upload file to storage using received SAS token uri
- Step 3: Verify status on upload
Only step 1 is different from the insert new dataset:
Prepare update operation
Prepare dataset update operation and get SAS URI for csv file upload Using the Veracity token from step 1
Payload
- StartAutomatically: If True, the job of processing the file will start automatically. However, if set to False, you need to start the job with api-endpoint. This is will trigger the job faster.
- Files can be more than one file. Operations: Upsert, Delete, Overwrite, Append. See operations defined below. Currently only one file is supported.
- DatasetId: id of dataset you want to update. can be retrieved from data workbench portal (see url on selected dataset)
Output Job id and sas_uri job_id is used to get status of the update after file is uploaded and job started
This is a PUT endpoint
Python
import requests
import json
#token from step 1
token = veracityToken
apiKey = "<api key>"
workspaceId = "<workspace id>"
datasetId = "<dataset id>"
targetFileName = "ImportData.csv"
base_url = "https://api.veracity.com/veracity/dw/gateway/api/v2"
endpoint = f"/workspaces/{workspaceId}/ingest/dataset/{datasetId}"
url = base_url + endpoint
headers = {
"Content-Type": "application/json",
"Ocp-Apim-Subscription-Key": apiKey,
"Authorization": f"Bearer {token}",
"User-Agent": "python-requests/2.31.0"
}
payload = {
"files": [
{
"fileName": targetFileName,
"operation": "Upsert"
}
],
"startAutomatically": True
}
try:
response = requests.put(url, json=payload, headers=headers)
response.raise_for_status()
except requests.exceptions.RequestException as e:
print(f"Error fetching sas uri: {e}")
result = response.json()
# Extract the SAS token
sas_uri = result.get('sasToken')
update_job_id = result.get('jobId')
C#
//output is a tuple with sasTokenUri and jobId
public async Task<(string, string)> GetSASUriForUpsertOperation(string datasetId,string targetFileName,
string workspaceId, string veracityToken, string subscriptionKey)
{
string sasTokenUri = string.Empty;
string job_Id = string.Empty;
string baseUrl = "https://api.veracity.com/veracity/dw/gateway/api/v2";
string url = $"{baseUrl}/workspaces/{workspaceId}/ingest/dataset/{datasetId}";
var token = veracityToken;
//Operations are Upsert, Overwrite, Delete or Append
var fileOperations = new List<Dictionary<string, object>>
{
new Dictionary<string, object>{
["fileName"] = targetFileName,
["operation"] = "Upsert"
}
};
//if start autoamtically = false, job must be started with seperate endpoint
var payload = new Dictionary<string, object>
{
{"startAutomatically", true },
{"files", fileOperations },
};
string jsonString = JsonConvert.SerializeObject(payload);
HttpContent content = new StringContent(jsonString, Encoding.UTF8, "application/json");
if (_httpClient == null)
_httpClient = new HttpClient();
_httpClient.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", subscriptionKey);
_httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
try
{
var result = await _httpClient.PutAsync(url, content);
if (result.IsSuccessStatusCode)
{
string json = result.Content.ReadAsStringAsync().Result;
using JsonDocument doc = JsonDocument.Parse(json);
JsonElement root = doc.RootElement;
JsonElement jobId = root.GetProperty("jobId");
JsonElement sasUri = root.GetProperty("sasToken");
sasTokenUri = sasUri.ToString().Trim('"');
job_Id = jobId.ToString().Trim('"');
}
}
catch (Exception ex)
{
throw ex;
}
return (sasTokenUri, job_Id);
}
Delete rows from dataset
"files": [
{
"fileName": "FileWithRowsToDelete.csv",
"operation": "Delete"
}
],
Multi operations
Updating multiple files with different operations will soon be supported.
"files": [
{
"fileName": "FileWithRowsToDelete.csv",
"operation": "Delete"
},
{
"fileName": "FileWithRowsToInsert.csv",
"operation": "Upsert"
}
],
Get schema version id
Schema version id must be used to upload a new dataset. Fetch schema id from url in Dataworkbench Portal (Schema management).
Python
apiKey = "<api key>"
workspaceId = "<Workspace id>"
schemaId = "<Schema id>"
base_url = "https://api.veracity.com/veracity/dw/gateway/api/v2"
endpoint = f"/workspaces/{workspaceId}/schemas/{schemaId}"
url = base_url + endpoint
headers = {
"Content-Type": "application/json",
"Ocp-Apim-Subscription-Key": apiKey,
"Authorization": f"Bearer {veracityToken}",
"User-Agent": "python-requests/2.31.0"
}
try:
response = requests.get(url, json=None, headers=headers)
response.raise_for_status()
except requests.exceptions.RequestException as e:
print(f"Error starting job: {e}")
result = response.json()
# Get the list of schema versions
schema_versions = result.get("schemaVersions", [])
# find version with isDefault == True
default_version = next(
(sv for sv in schema_versions if sv.get("isDefault") is True),
None
)
if default_version:
print("Default version ID:", default_version["id"])
else:
print("No default schema version found.")
C#
public async Task<string> GetSchemaVersionId(string schemaId, string workspaceId, string veracityToken, string subscriptionKey)
{
string schemaVersionId = null;
if (!string.IsNullOrEmpty(schemaId))
{
string baseUrl = "https://api.veracity.com/veracity/dw/gateway/api/v2";
string url = $"{baseUrl}/workspaces/{workspaceId}/schemas/{schemaId}";
var token = veracityToken;
if (_httpClient == null)
_httpClient = new HttpClient();
_httpClient.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", subscriptionKey);
_httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
try
{
var result = await _httpClient.GetAsync(url);
if (result.IsSuccessStatusCode)
{
string json = result.Content.ReadAsStringAsync().Result;
using JsonDocument doc = JsonDocument.Parse(json);
JsonElement root = doc.RootElement;
//in this example, reading first version
JsonElement schemaVersions = root.GetProperty("schemaVersions")[0];
var id = schemaVersions.GetProperty("id");
return id.ToString();
}
}
catch (Exception ex)
{
}
}
return schemaVersionId;
}