Workflow

The enables the Data Platform to invoke jobs defined in Databricks without entering the Analytics environment. A typical use case is processing ingested data....

The Workflow API enables the Data Platform to invoke jobs defined in Databricks without entering the Analytics environment. A typical use case is processing ingested data. In Azure Databricks, a job automates tasks such as running notebooks, JARs, Python scripts, or workflows. A job can have multiple tasks and can be executed on a schedule or on demand.

See Databricks documentation for how to create jobs in Databricks.

API endpoints

To browse the API, see the Analytics section in the API Explorer.

Base URL: https://api.veracity.com/veracity/dw/gateway/api/v2

Authentication: Authentication and a subscription key are required for each API request.

Only Data Workbench admins have permission to call the Workflow API.

List all job definitions

You can list all jobs available in the Databricks workspace connected to the Data Workbench workspace. When a job definition is created in Databricks, it is not automatically shared with all workspace users, so other users may not see it in Databricks. The API endpoint lists all job definitions, including jobs that are not shared with every user.

GET: https://api.veracity.com/veracity/dw/gateway/api/v2/workspaces/{workspaceId}/jobs

workspaceId is the Data Workbench workspace ID. It must be a UUID value.

The response contains a jobs array with job definitions from the Databricks workspace related to the Data Workbench workspace. If the response is paged, use the returned page token with the optional pageToken query parameter.

{
  "jobs": [
    {
      "jobId": 0,
      "jobName": "string",
      "description": "string",
      "tags": {},
      "creatorUserName": "user@example.com",
      "createdTime": "2025-12-15T08:35:52.282+00:00",
      "parameters": [
        {
          "name": "InputDataset",
          "default": "string"
        }
      ]
    }
  ],
  "nextPageToken": "string",
  "prevPageToken": "string"
}

Run a job

You can start, or invoke, a Databricks job from the API.

To run a job without entering Databricks, use the jobId from the job definition and call the following endpoint:

POST: https://api.veracity.com/veracity/dw/gateway/api/v2/workspaces/{workspaceId}/jobs/{jobId}/invoke

jobId is the Databricks job ID returned by the job definition list. It is an integer value.

The request body contains values for the job parameters. In the following example, the job has two parameters: InputDataset and ResultDatasetName.

{
  "InputDataset": "turbinedata2023",
  "ResultDatasetName": "aggregateddata"
}

The notebook or script must know how to interpret these parameter values.

The API starts the job run. Use the returned runId to check the run status or cancel the run.

Get status of a job run

Use the runId returned when the job was invoked to request the status of the run.

GET: https://api.veracity.com/veracity/dw/gateway/api/v2/workspaces/{workspaceId}/runs/{runId}/status

runId is the Databricks job run ID. It is an integer value.

Cancel a run

Use the runId returned when the job was invoked to cancel that run.

POST: https://api.veracity.com/veracity/dw/gateway/api/v2/workspaces/{workspaceId}/runs/{runId}/cancel

Cluster

Jobs run on clusters, which can be:

  • Job clusters: clusters created for the job and terminated after completion.
  • All-purpose clusters: shared clusters used for interactive work and jobs.