Skip to main content

Requirements

definity monitors Databricks serverless jobs using a read-only connection to your workspace. Setup is done once and covers every job in the workspace.

There is no

  • External agent installation in your environment
  • Pipeline code change
  • Per-job configuration

Three Databricks roles are involved – described in the steps below:

Prerequisites – account admin​

1. Create a service principal – workspace admin​

Create a Databricks service principal for definity and generate OAuth credentials for it: a client ID and a client secret. Databricks shows the secret once, at creation.

definity authenticates as this principal, so access does not depend on an individual user. The principal only reads – it never writes data, starts or edits jobs, or creates compute.

2. Grant read-only access – metastore admin​

Run the grants as the metastore owner or an account admin. A workspace admin cannot grant on the system catalog; it fails with User does not have MANAGE on Schema 'system.lakeflow'.

Replace <principal> with the principal's application ID – the UUID, not the numeric id:

GRANT USE CATALOG ON CATALOG system TO `<principal>`;
GRANT USE SCHEMA, SELECT ON SCHEMA system.lakeflow TO `<principal>`;
GRANT USE SCHEMA, SELECT ON SCHEMA system.query    TO `<principal>`;
GRANT USE SCHEMA, SELECT ON SCHEMA system.access   TO `<principal>`;
GRANT USE SCHEMA, SELECT ON SCHEMA system.compute  TO `<principal>`;

-- once per catalog your pipelines write to
GRANT BROWSE ON CATALOG <catalog> TO `<principal>`;
GrantCovers
system.lakeflowJob, run and task structure, timings and status.
system.querySQL statement text, I/O and duration.
system.accessTable- and path-level lineage.
system.computeCompute type per task.
BROWSE on a catalogDataset column schemas via information_schema – metadata only, no data access.

BROWSE is the least-privilege grant for reading a catalog's information_schema. Datasets addressed by path (s3://…) get lineage without column schemas.

3. Grant the Workspace access entitlement – workspace admin​

definity's queries run on serverless compute, which requires this entitlement. A service principal created through the workspace UI does not get it by default.

  1. Click your username in the top-right corner, then Settings.
  2. Open the Identity and access tab.
  3. Next to Service principals, click Manage.
  4. Click the principal – add it with Add service principal if it is not listed.
  5. Turn on the Workspace access toggle.

No other entitlement is needed.

4. Connection details​

Send definity:

ValueNotes
Workspace URLhttps://<your-workspace>.cloud.databricks.com
Client IDThe service principal's application ID (a UUID).
Client secretThe OAuth secret from step 1.
SQL warehouse IDOptional – see below.

SQL warehouse. Queries run on serverless compute by default and need no warehouse. Provide a warehouse ID only if serverless compute is unavailable in the workspace, or if you prefer the queries to run on a warehouse. Grant the principal CAN_USE on it, or collection fails with PERMISSION_DENIED: You do not have permission to use the SQL Warehouse.

5. Network access​

definity reaches your workspace over HTTPS: the REST APIs and the serverless endpoint, or the warehouse endpoint if you provide a warehouse. If the workspace enforces an IP access list, allowlist definity's egress IP ranges, which definity provides. No inbound access to your network is required.

Coverage​

  • Covered here – serverless jobs.
  • Covered using a different agent – classic (cluster) compute and SQL warehouse tasks; use the Spark agent.
  • Skipped – orchestration-only tasks (run_job, if/else, for_each), which carry no work of their own.
Last updated on