Schema management
Datasets stored in a workspace in Data Workbench must be based on a schema. A schema is like the definition of a table defining the columns of the schema....
Datasets stored in a workspace in Data Workbench must be based on a schema. A schema is like the definition of a table defining the columns of the schema. Schemas can be created and managed in the portal or using APIs. You need to be Admin in workspace to modify a schema.
Versioning
A schema can have several versions. When creating a new schema, a first schema version is autoamtically created. Only one version can be active at one time.
API endpoints
To browse the Data platform gateway api, see section Schemas.
All response formats and status codes are documented in the API Explorer.
Base url: https://api.veracity.com/veracity/dw/gateway/api/v2
Authentication: Authentication and subscription key is required for each api request, see details..
Create a new schema
In the example below a schema is created with two columns. To define "rowValidators" in input payload - Validator must be subcribed to (Validation Rule Management).
output Schema created with schema id and a schema version id. When creating a new schema, the active schema Version 1 will also be created and the version name is '[Schema name] V1'. Note: When uploading a new dataset, schema version id is used.
Configure each column by setting:
- isSortable: Enable if you want users to be able to sort the data by this column.
- isFilterable: Enable if you want users to be able to filter the data based on this column's values.
- isRequired: Enable if the column must have a value for every row in the dataset. This is a critical validation rule since missing a required column will cause validation to fail.
Python
apiKey = "<my api key>"
workspaceId = "<workspace id>"
base_url = "https://api.veracity.com/veracity/dw/gateway/api/v2"
endpoint = f"/workspaces/{workspaceId}/schemas/add"
url = base_url + endpoint
headers = {
"Content-Type": "application/json",
"Ocp-Apim-Subscription-Key": apiKey,
"Authorization": f"Bearer {veracityToken}",
"User-Agent": "python-requests/2.31.0"
}
payload = {
"name": "TestSchema2",
"shortName": "Test",
"description": "some description",
"columns": [
{
"name": "TS",
"displayName": "Timestamp",
"type": "DateTime",
"format": "YYYY-MM-DDThh:mm:ss",
"filterType": "List",
"filterTypes": ["List"],
"isRequired": True,
"validations": None,
"metaType": "Validation",
"isSortable": True,
"description": "string",
"isFilterable": True,
"isKey": True,
"order": 0
},
{
"name": "Temp",
"displayName": "Temperature",
"type": "Decimal",
"filterType": "List",
"filterTypes": ["List"],
"isRequired": True,
"validations": None,
"metaType": "Validation",
"isSortable": True,
"description": "temp",
"isFilterable": False,
"isKey": False,
"order": 1
}
],
"rowValidators": None
}
try:
response = requests.post(url, json=payload, headers=headers)
response.raise_for_status()
result = response.json()
schemaId = result.get("id")
schemaVersionId = result.get("schemaVersions")[0].get("id")
except requests.exceptions.RequestException as e:
print(f"Error creating a schema: {e}")
Rule validation To set validation rules, subscription to Rule validation is required.
"validations": [
{
"severity": "Error",
"validatorId": "string"
}
],
"metaType": "Validation",
Data types and format
A column needs a datatype. We support the following:
- String
- Boolean
- Datetime
- Date Only
- Decimal
- Float
- Double
- Integer 32
- Integer64
Format is used by some of the datatypes:
- Date only: Default format is YYYY-MM-DD
- Datetime: Default format is YYYY-MM-DDThh:mm:ss
- Decimal: Allows for specifying Precision (total number of digits) and Scale (number of digits after decimal)
Below is a practical recommendation for when to use each numeric datatype in datasets. These guidelines align with common data-engineering best practices for analytics, cost efficiency, and performance.
Decimal
Decimals store numbers exactly—no floating‑point drift. When defining a column with decimals; you can configure precision and scale. Default values are:
- Precision: 28 (total no of digits before and after decimal)
- Scale: 8 (total no of digits after decimal)
Use when you neeed exact precision (no rounding errors). Values represent money, percentages, ratios, or measurements requiring accuracy. Downstream analytics depend on exact values (e.g., financial reporting, regulatory outputs).
Examples: Energy prices (e.g., 125.76 USD/MWh) Financial transactions Precision readings (e.g., efficiency factors)
Double
Follow the IEEE 754 floating‑point standard, which stores numbers in binary and allows fast calculations—but with rounding errors.
- ~15–16 decimal digits of precision (total no of digits before and after decimal)
- Much more accurate than float
- Still approximate, but errors are far smaller
Float
Follow the IEEE 754 floating‑point standard, which stores numbers in binary and allows fast calculations—but with rounding errors. Float is efficient, compact, and suitable for large volumes of numeric telemetry. To be used when you need fast computations and approximate values are acceptable.
Values can contain many decimal places but don’t require financial‑grade accuracy. Used in scientific calculations, sensor data, simulation outputs, or telemetry.
- ~7 decimal digits of precision (total no of digits before and after decimal)
- Higher risk of rounding errors
- Less accurate for large or very precise numbers
Examples: Sensor temperatures (21.34567 °C) Power outputs from simulations Derived metrics where slight rounding noise is acceptable
Integer32 (Int32)
Int32 is smaller and faster than Int64 when the range is safe.
Use when values are whole numbers that fit within –2.1B to +2.1B, which is enough for most operational datasets. You want an efficient, compact type.
Examples: Status codes Counts of events Limited range IDs (e.g., turbine number, device ID under a few million) Year, day, or other discrete values
Integer64 (Int64)
Int64 avoids overflow and supports very large integer values safely.
Use when whole numbers exceed the Int32 range. You work with large sequences or identifiers. Timestamps represented as Unix time (ms) or ticks. Large cumulative counters, e.g., energy produced over lifetime.
Examples: Large unique IDs (GUID-like integers, surrogate keys) Unix timestamps in milliseconds (can be > 2.1B) High-volume meter readings or incremental counters Row counts or cumulative sums
Filter and filter types
Specifying that a column is filterable, allows data to be queried with filterd based on values in that column. The following filter types are defined: Note: Currently not all filter types are supported for dataset query
| Data type | Filtertypes |
|---|---|
| String | List, StringContains, Equals |
| Decimal, Float, Double, Integrer | Equals,Greater,GreaterOrEqual,Less,LessOrEqual |
| DateTime, Date only | Greater,GreaterOrEqual,Less,LessOrEqual |
| Boolean | Equals |
Note NonFromList not supported
Modify schema
After a schema version is locked you can only change columns by creating a new schema version, see add schemaversion.
When modifying a schema, you can set new name, description and short name using Patch endpoint. To change the columns, you need to either add new schema version or modify columns in an existing schema version.
Patch changes:
Python
apiKey = "<api key>"
workspaceId = "<workspace id>"
schema_id = "<schema id>"
base_url = "https://api.veracity.com/veracity/dw/gateway/api/v2"
endpoint = f"/workspaces/{workspaceId}/schemas/{schema_id}"
url = base_url + endpoint
headers = {
"Content-Type": "application/json",
"Ocp-Apim-Subscription-Key": apiKey,
"Authorization": f"Bearer {veracityToken}",
"User-Agent": "python-requests/2.31.0"
}
payload = {
"name": "new schema name",
"description": "new description",
"shortName": "Test100"
}
try:
response = requests.patch(url, json=payload, headers=headers)
response.raise_for_status()
result = response.json()
except requests.exceptions.RequestException as e:
print(f"Error updating schema: {e}")
Add new schema version
In this example, a new version of previous schema is created where a new column is added.
Python
apiKey = "<api key>"
workspaceId = "<workspace id>"
schema_id = "<schema id>"
base_url = "https://api.veracity.com/veracity/dw/gateway/api/v2"
endpoint = f"/workspaces/{workspaceId}/schemas/schemaversions/add"
url = base_url + endpoint
headers = {
"Content-Type": "application/json",
"Ocp-Apim-Subscription-Key": apiKey,
"Authorization": f"Bearer {veracityToken}",
"User-Agent": "python-requests/2.31.0"
}
payload = {
"schemaId": schema_id,
"schemaVersionName": "Version 2",
"description": "some description of version",
"columns": [
{
"name": "TS",
"displayName": "Timestamp",
"type": "DateTime",
"format": "YYYY-MM-DDThh:mm:ss",
"isRequired": True,
"isSortable": True,
"description": "time stamp",
"isFilterable": True,
"isKey": True,
"order": 0
},
{
"name": "Temp",
"displayName": "Temperature",
"type": "Decimal",
"isRequired": True,
"isSortable": True,
"description": "temp",
"isFilterable": True,
"isKey": False,
"order": 1
},
{
"name": "Wind",
"displayName": "Wind",
"type": "Decimal",
"isRequired": True,
"isSortable": True,
"description": "wind",
"isFilterable": True,
"isKey": False,
"order": 2
}
]
}
try:
response = requests.post(url, json=payload, headers=headers)
response.raise_for_status()
result = response.json()
schema_version_id = result.get("id")
except requests.exceptions.RequestException as e:
print(f"Error updating schema: {e}")
Modify schema version
This is a PATCH operation. In this example, a column is removed from the schema_version from previous example.
Python
apiKey = "<api key>"
workspaceId = "<workspace id>"
schema_id = "<schema id>"
schema_version_id = "<schema version id>"
base_url = "https://api.veracity.com/veracity/dw/gateway/api/v2"
endpoint = f"/workspaces/{workspaceId}/schemas/{schema_id}/schemaversions/{schema_version_id}"
url = base_url + endpoint
headers = {
"Content-Type": "application/json",
"Ocp-Apim-Subscription-Key": apiKey,
"Authorization": f"Bearer {veracityToken}",
"User-Agent": "python-requests/2.31.0"
}
payload = {
"schemaId": schema_id,
"schemaVersionName": "Version 2",
"description": "some description of version",
"columns": [
{
"name": "TS",
"displayName": "Timestamp",
"type": "DateTime",
"format": "YYYY-MM-DDThh:mm:ss",
"isRequired": True,
"isSortable": True,
"description": "time stamp new desc",
"isFilterable": True,
"isKey": True,
"order": 0
}, {
"name": "Wind",
"displayName": "Wind",
"type": "Decimal",
"isRequired": True,
"isSortable": True,
"description": "wind",
"isFilterable": True,
"isKey": False,
"order": 1
}
]
}
try:
response = requests.patch(url, json=payload, headers=headers)
response.raise_for_status()
result = response.json()
schema_version_id = result.get("id")
except requests.exceptions.RequestException as e:
print(f"Error updating schema: {e}")
Set schema versions active
A schema can have one active version (i.e. the default version). When using Validator, this is the schema being used.
Python
apiKey = "<api key>"
workspaceId = "<workspace id>"
schema_id = "<schema id>"
schema_version_id = "<schema version id>"
base_url = "https://api.veracity.com/veracity/dw/gateway/api/v2"
endpoint = f"/workspaces/{workspaceId}/schemas/{schema_id}/schemaversions/{schema_version_id}/activate"
url = base_url + endpoint
headers = {
"Content-Type": "application/json",
"Ocp-Apim-Subscription-Key": apiKey,
"Authorization": f"Bearer {veracityToken}",
"User-Agent": "python-requests/2.31.0"
}
try:
response = requests.post(url, json=None, headers=headers)
response.raise_for_status()
result = response.json()
print(result)
except requests.exceptions.RequestException as e:
print(f"Error activating schema: {e}")
Lock schema version
A locked schema version can not be modified with a PATCH and a new schema version would need to be created if a change is required.
Python
apiKey = "<api key>"
workspaceId = "<workspace id>"
schema_id = "<schema id>"
schema_version_id = "<schema version id>"
base_url = "https://api.veracity.com/veracity/dw/gateway/api/v2"
endpoint = f"/workspaces/{workspaceId}/schemas/{schema_id}/schemaversions/{schema_version_id}/lock"
url = base_url + endpoint
headers = {
"Content-Type": "application/json",
"Ocp-Apim-Subscription-Key": apiKey,
"Authorization": f"Bearer {veracityToken}",
"User-Agent": "python-requests/2.31.0"
}
try:
response = requests.post(url, json=None, headers=headers)
response.raise_for_status()
result = response.getstatus_code()
print(result)
except requests.exceptions.RequestException as e:
print(f"Error locking schema: {e}")