Schema management

Datasets stored in a workspace in Data Workbench must be based on a schema. A schema is like the definition of a table defining the columns of the schema....

Datasets stored in a workspace in Data Workbench must be based on a schema. A schema is like the definition of a table defining the columns of the schema. Schemas can be created and managed in the portal or using APIs. You need to be Admin in workspace to modify a schema.

Versioning

A schema can have several versions. When creating a new schema, a first schema version is autoamtically created. Only one version can be active at one time.

API endpoints

To browse the Data platform gateway api, see section Schemas.

All response formats and status codes are documented in the API Explorer.

Base url: https://api.veracity.com/veracity/dw/gateway/api/v2

Authentication: Authentication and subscription key is required for each api request, see details..

Create a new schema

In the example below a schema is created with two columns. To define "rowValidators" in input payload - Validator must be subcribed to (Validation Rule Management).

output Schema created with schema id and a schema version id. When creating a new schema, the active schema Version 1 will also be created and the version name is '[Schema name] V1'. Note: When uploading a new dataset, schema version id is used.

Configure each column by setting:

  • isSortable: Enable if you want users to be able to sort the data by this column.
  • isFilterable: Enable if you want users to be able to filter the data based on this column's values.
  • isRequired: Enable if the column must have a value for every row in the dataset. This is a critical validation rule since missing a required column will cause validation to fail.

Python

apiKey =  "<my api key>"
workspaceId = "<workspace id>"

base_url = "https://api.veracity.com/veracity/dw/gateway/api/v2"
endpoint = f"/workspaces/{workspaceId}/schemas/add"
url = base_url + endpoint

headers = {
    "Content-Type": "application/json",
    "Ocp-Apim-Subscription-Key": apiKey,
    "Authorization": f"Bearer {veracityToken}",
    "User-Agent": "python-requests/2.31.0"
}

payload = {
    "name": "TestSchema2",
    "shortName": "Test",
    "description": "some description",
    "columns": [
        {
            "name": "TS",
            "displayName": "Timestamp",
            "type": "DateTime",
            "format": "YYYY-MM-DDThh:mm:ss",
            "filterType": "List",
            "filterTypes": ["List"],
            "isRequired": True,
            "validations": None,
            "metaType": "Validation",
            "isSortable": True,
            "description": "string",
            "isFilterable": True,
            "isKey": True,
            "order": 0
        },
        {
            "name": "Temp",
            "displayName": "Temperature",
            "type": "Decimal",
            "filterType": "List",
            "filterTypes": ["List"],
            "isRequired": True,
            "validations": None,
            "metaType": "Validation",
            "isSortable": True,
            "description": "temp",
            "isFilterable": False,
            "isKey": False,
            "order": 1
        }
    ],
    "rowValidators": None
}
try:
    response = requests.post(url, json=payload, headers=headers)
    response.raise_for_status()   
    result = response.json()
    schemaId = result.get("id")
    schemaVersionId = result.get("schemaVersions")[0].get("id")

except requests.exceptions.RequestException as e:
    print(f"Error creating a schema: {e}")

Rule validation To set validation rules, subscription to Rule validation is required.

"validations": [
   {
          "severity": "Error",
          "validatorId": "string"
   }
],      
"metaType": "Validation",

Data types and format

A column needs a datatype. We support the following:

  • String
  • Boolean
  • Datetime
  • Date Only
  • Decimal
  • Float
  • Double
  • Integer 32
  • Integer64

Format is used by some of the datatypes:

  • Date only: Default format is YYYY-MM-DD
  • Datetime: Default format is YYYY-MM-DDThh:mm:ss
  • Decimal: Allows for specifying Precision (total number of digits) and Scale (number of digits after decimal)

Below is a practical recommendation for when to use each numeric datatype in datasets. These guidelines align with common data-engineering best practices for analytics, cost efficiency, and performance.

Decimal

Decimals store numbers exactly—no floating‑point drift. When defining a column with decimals; you can configure precision and scale. Default values are:

  • Precision: 28 (total no of digits before and after decimal)
  • Scale: 8 (total no of digits after decimal)

Use when you neeed exact precision (no rounding errors). Values represent money, percentages, ratios, or measurements requiring accuracy. Downstream analytics depend on exact values (e.g., financial reporting, regulatory outputs).

Examples: Energy prices (e.g., 125.76 USD/MWh) Financial transactions Precision readings (e.g., efficiency factors)

Double

Follow the IEEE 754 floating‑point standard, which stores numbers in binary and allows fast calculations—but with rounding errors.

  • ~15–16 decimal digits of precision (total no of digits before and after decimal)
  • Much more accurate than float
  • Still approximate, but errors are far smaller

Float

Follow the IEEE 754 floating‑point standard, which stores numbers in binary and allows fast calculations—but with rounding errors. Float is efficient, compact, and suitable for large volumes of numeric telemetry. To be used when you need fast computations and approximate values are acceptable.

Values can contain many decimal places but don’t require financial‑grade accuracy. Used in scientific calculations, sensor data, simulation outputs, or telemetry.

  • ~7 decimal digits of precision (total no of digits before and after decimal)
  • Higher risk of rounding errors
  • Less accurate for large or very precise numbers

Examples: Sensor temperatures (21.34567 °C) Power outputs from simulations Derived metrics where slight rounding noise is acceptable

Integer32 (Int32)

Int32 is smaller and faster than Int64 when the range is safe.

Use when values are whole numbers that fit within –2.1B to +2.1B, which is enough for most operational datasets. You want an efficient, compact type.

Examples: Status codes Counts of events Limited range IDs (e.g., turbine number, device ID under a few million) Year, day, or other discrete values

Integer64 (Int64)

Int64 avoids overflow and supports very large integer values safely.

Use when whole numbers exceed the Int32 range. You work with large sequences or identifiers. Timestamps represented as Unix time (ms) or ticks. Large cumulative counters, e.g., energy produced over lifetime.

Examples: Large unique IDs (GUID-like integers, surrogate keys) Unix timestamps in milliseconds (can be > 2.1B) High-volume meter readings or incremental counters Row counts or cumulative sums

Filter and filter types

Specifying that a column is filterable, allows data to be queried with filterd based on values in that column. The following filter types are defined: Note: Currently not all filter types are supported for dataset query

Data type Filtertypes
String List, StringContains, Equals
Decimal, Float, Double, Integrer Equals,Greater,GreaterOrEqual,Less,LessOrEqual
DateTime, Date only Greater,GreaterOrEqual,Less,LessOrEqual
Boolean Equals

Note NonFromList not supported

Modify schema

After a schema version is locked you can only change columns by creating a new schema version, see add schemaversion.

When modifying a schema, you can set new name, description and short name using Patch endpoint. To change the columns, you need to either add new schema version or modify columns in an existing schema version.

Patch changes:

Python

apiKey =  "<api key>"
workspaceId = "<workspace id>"
schema_id = "<schema id>"

base_url = "https://api.veracity.com/veracity/dw/gateway/api/v2"
endpoint = f"/workspaces/{workspaceId}/schemas/{schema_id}"
url = base_url + endpoint

headers = {
    "Content-Type": "application/json",
    "Ocp-Apim-Subscription-Key": apiKey,
    "Authorization": f"Bearer {veracityToken}",
    "User-Agent": "python-requests/2.31.0"
}

payload = {
  "name": "new schema name",
  "description": "new description",
  "shortName": "Test100"
}
try:
    response = requests.patch(url, json=payload, headers=headers)
    response.raise_for_status()   
    result = response.json()   
except requests.exceptions.RequestException as e:
    print(f"Error updating schema: {e}")

Add new schema version

In this example, a new version of previous schema is created where a new column is added.

Python

apiKey =  "<api key>"
workspaceId = "<workspace id>"
schema_id = "<schema id>"

base_url = "https://api.veracity.com/veracity/dw/gateway/api/v2"
endpoint = f"/workspaces/{workspaceId}/schemas/schemaversions/add"
url = base_url + endpoint

headers = {
    "Content-Type": "application/json",
    "Ocp-Apim-Subscription-Key": apiKey,
    "Authorization": f"Bearer {veracityToken}",
    "User-Agent": "python-requests/2.31.0"
}

payload = {  
  "schemaId": schema_id,
  "schemaVersionName": "Version 2",
  "description": "some description of version",
  "columns": [
    {
      "name": "TS",
      "displayName": "Timestamp",
      "type": "DateTime",
      "format": "YYYY-MM-DDThh:mm:ss",
      "isRequired": True,  
      "isSortable": True,
      "description": "time stamp",
      "isFilterable": True,
      "isKey": True,
      "order": 0      
    },
    {
      "name": "Temp",
      "displayName": "Temperature",
      "type": "Decimal",
      "isRequired": True,  
      "isSortable": True,
      "description": "temp",
      "isFilterable": True,
      "isKey": False,
      "order": 1      
    },
    {
      "name": "Wind",
      "displayName": "Wind",
      "type": "Decimal",
      "isRequired": True,  
      "isSortable": True,
      "description": "wind",
      "isFilterable": True,
      "isKey": False,
      "order": 2      
    }
  ]  
}
try:
    response = requests.post(url, json=payload, headers=headers)
    response.raise_for_status()   
    result = response.json()   
    schema_version_id = result.get("id")
except requests.exceptions.RequestException as e:
    print(f"Error updating schema: {e}")

Modify schema version

This is a PATCH operation. In this example, a column is removed from the schema_version from previous example.

Python

apiKey =  "<api key>"
workspaceId = "<workspace id>"
schema_id = "<schema id>"
schema_version_id = "<schema version id>"

base_url = "https://api.veracity.com/veracity/dw/gateway/api/v2"
endpoint = f"/workspaces/{workspaceId}/schemas/{schema_id}/schemaversions/{schema_version_id}"
url = base_url + endpoint

headers = {
    "Content-Type": "application/json",
    "Ocp-Apim-Subscription-Key": apiKey,
    "Authorization": f"Bearer {veracityToken}",
    "User-Agent": "python-requests/2.31.0"
}

payload = {  
  "schemaId": schema_id,
  "schemaVersionName": "Version 2",
  "description": "some description of version",
  "columns": [
    {
      "name": "TS",
      "displayName": "Timestamp",
      "type": "DateTime",
      "format": "YYYY-MM-DDThh:mm:ss",
      "isRequired": True,  
      "isSortable": True,
      "description": "time stamp new desc",
      "isFilterable": True,
      "isKey": True,
      "order": 0      
    },  {
      "name": "Wind",
      "displayName": "Wind",
      "type": "Decimal",
      "isRequired": True,  
      "isSortable": True,
      "description": "wind",
      "isFilterable": True,
      "isKey": False,
      "order": 1      
    }
  ]  
}
try:
    response = requests.patch(url, json=payload, headers=headers)
    response.raise_for_status()   
    result = response.json()   
    schema_version_id = result.get("id")
  
except requests.exceptions.RequestException as e:
    print(f"Error updating schema: {e}")

Set schema versions active

A schema can have one active version (i.e. the default version). When using Validator, this is the schema being used.

Python

apiKey =  "<api key>"
workspaceId = "<workspace id>"
schema_id = "<schema id>"
schema_version_id = "<schema version id>"
base_url = "https://api.veracity.com/veracity/dw/gateway/api/v2"
endpoint = f"/workspaces/{workspaceId}/schemas/{schema_id}/schemaversions/{schema_version_id}/activate"
url = base_url + endpoint

headers = {
    "Content-Type": "application/json",
    "Ocp-Apim-Subscription-Key": apiKey,
    "Authorization": f"Bearer {veracityToken}",
    "User-Agent": "python-requests/2.31.0"
}

try:
    response = requests.post(url, json=None, headers=headers)
    response.raise_for_status()   
    result = response.json()   
    print(result)
  
except requests.exceptions.RequestException as e:
    print(f"Error activating schema: {e}")

Lock schema version

A locked schema version can not be modified with a PATCH and a new schema version would need to be created if a change is required.

Python

apiKey =  "<api key>"
workspaceId = "<workspace id>"
schema_id = "<schema id>"
schema_version_id = "<schema version id>"

base_url = "https://api.veracity.com/veracity/dw/gateway/api/v2"
endpoint = f"/workspaces/{workspaceId}/schemas/{schema_id}/schemaversions/{schema_version_id}/lock"
url = base_url + endpoint

headers = {
    "Content-Type": "application/json",
    "Ocp-Apim-Subscription-Key": apiKey,
    "Authorization": f"Bearer {veracityToken}",
    "User-Agent": "python-requests/2.31.0"
}

try:
    response = requests.post(url, json=None, headers=headers)
    response.raise_for_status()   
    result = response.getstatus_code()
    print(result)
  
except requests.exceptions.RequestException as e:
    print(f"Error locking schema: {e}")