Architecture
Overviewβ
Definity is an enterprise lakehouse optimization and observability platform. It connects to the enterprise data lakehouse and allows the data teams to monitor, optimize and protect their data infrastructure.
Definity is delivered as SaaS, hosted in Definity's secure cloud environment. Metrics & metadata are reported to Definity servers.
For all deployment models and their security controls, see Deployment Models in the Trust Center.
Componentsβ
Agentsβ
Definity uses dedicated data pipeline agents that are added by configuration to each pipeline in the customer's data platform. The agents are provided as a code library (JAR File / Python library) and the agent code runs as part of the customer's data application. The agents communicate with the Definity platform server.
The agents report each run to the server and apply the actions it sends back, such as Auto-Tune configuration changes (see Data Flow Summary). The server analyzes every run, decides which optimizations and interventions to apply, and exposes a web application for the pipeline owners.
Agent instrumentationβ
Definity agents are added to the data pipeline and run as part of the pipeline. Specific instrumentation info can be found in the relevant installation section.
Serverβ
Definity's platform server is hosted and operated in Definity's cloud. All communications with the agents and the web application are TLS encrypted and authenticated with OAuth token.
Zero-Code Change integrationβ
Definity achieves zero code change instrumentation via agent-based instrumentation:
-
The agent automatically hooks into pipeline runtimes (e.g., Spark, DBT) at execution time.
-
It extracts execution context, schema evolution, and metric data from runtime metadata.
-
No modifications to job code, SQL, or DAGs are required.
Data Flow Summaryβ
-
Pipeline job starts in customer environment.
-
Definity agent attaches at runtime and observes job metadata.
-
Metadata and metrics are transmitted via TLS to Definityβs control plane.
-
The server analyzes each run and sends back actions, such as Auto-Tune configuration changes, and surfaces the results in the web application and APIs.
From agent to serverβ
| Data Type | Notes |
|---|---|
| Pipeline Metadata | Job name, start/stop time, run status, DAG/task IDs |
| Schema Metadata | Column names, data types, evolution history |
| Execution metrics | Duration, counts (including row counts), error rates, resource usage |
| Column distribution statistics | Value histograms and other per-column statistics |
| Diagnostic samples | A few key values from the slowest tasks and the row a failed task was on (Spark) |
| Query text | SQL and plans, including literals |
No bulk data or query results leave the customer's environment.
From server to agentβ
The agent opens every connection, outbound over TLS. The server sends instructions back only as responses on those connections: it never connects into your network, and nothing listens for inbound connections on the agent side. The agent can act on these instructions:
| From server to agent | Purpose | Where to see it |
|---|---|---|
| Spark configuration overrides at task start | Auto-Tune | The task run's Params tab (autotune.-prefixed rows) and the job's Auto-tune activity. See Review the changes Auto-Tune applied |
| Output diversion plan | Side-by-side validation runs | β |
| Stop request | Break test action | β |
| Databricks API calls from inside the job | Auto-Tune | Your Databricks audit log (system.access.audit), under the identity the agent calls with |
| Data quality metric requests | Opt-in column metrics: extra queries, or Spark Observe on running ones | β |
The server can request only these predefined operations; it cannot send the agent code or scripts to run.
If the agent loses contact with the server, or hits an error, a timeout or excess overhead, it disables itself and the job keeps running without it. No further instructions are applied to that run.
Every agent call to the server is logged on the server side.
Supported Runtimes and Targetsβ
| Runtime / Platform | Support | Notes |
|---|---|---|
| Apache Spark (Kubernetes / YARN) | β Full support | Versions & Details |
| Google Dataproc (Cluster & Serverless) | β Full support | Integrated with GCP IAM and Dataproc APIs. Versions & Details |
| Databricks | β Full support | Versions & Details |
| Amazon EMR (Cluster & Serverless) | β Full support | Versions & Details |
| Others | π§© Roadmap | Additional runtimes under evaluation. |
Securityβ
Security, privacy and compliance are covered in the Trust Center.