Skip to main content

Architecture

Overview​

Definity is an enterprise lakehouse optimization and observability platform. It connects to the enterprise data lakehouse and allows the data teams to monitor, optimize and protect their data infrastructure.

Definity is delivered as SaaS, hosted in Definity's secure cloud environment. Metrics & metadata are reported to Definity servers.

For all deployment models and their security controls, see Deployment Models in the Trust Center.

Components​

Agents​

Definity uses dedicated data pipeline agents that are added by configuration to each pipeline in the customer's data platform. The agents are provided as a code library (JAR File / Python library) and the agent code runs as part of the customer's data application. The agents communicate with the Definity platform server.

The agents report each run to the server and apply the actions it sends back, such as Auto-Tune configuration changes (see Data Flow Summary). The server analyzes every run, decides which optimizations and interventions to apply, and exposes a web application for the pipeline owners.

Agent instrumentation​

Definity agents are added to the data pipeline and run as part of the pipeline. Specific instrumentation info can be found in the relevant installation section.

Server​

Definity's platform server is hosted and operated in Definity's cloud. All communications with the agents and the web application are TLS encrypted and authenticated with OAuth token.

SaaS deployment: agents connect outbound to the definity cloud; no inbound connections.

Zero-Code Change integration​

Definity achieves zero code change instrumentation via agent-based instrumentation:

  • The agent automatically hooks into pipeline runtimes (e.g., Spark, DBT) at execution time.

  • It extracts execution context, schema evolution, and metric data from runtime metadata.

  • No modifications to job code, SQL, or DAGs are required.

Data Flow Summary​

  1. Pipeline job starts in customer environment.

  2. Definity agent attaches at runtime and observes job metadata.

  3. Metadata and metrics are transmitted via TLS to Definity’s control plane.

  4. The server analyzes each run and sends back actions, such as Auto-Tune configuration changes, and surfaces the results in the web application and APIs.

From agent to server​

Data TypeNotes
Pipeline MetadataJob name, start/stop time, run status, DAG/task IDs
Schema MetadataColumn names, data types, evolution history
Execution metricsDuration, counts (including row counts), error rates, resource usage
Column distribution statisticsValue histograms and other per-column statistics
Diagnostic samplesA few key values from the slowest tasks and the row a failed task was on (Spark)
Query textSQL and plans, including literals

No bulk data or query results leave the customer's environment.

From server to agent​

The agent opens every connection, outbound over TLS. The server sends instructions back only as responses on those connections: it never connects into your network, and nothing listens for inbound connections on the agent side. The agent can act on these instructions:

From server to agentPurposeWhere to see it
Spark configuration overrides at task startAuto-TuneThe task run's Params tab (autotune.-prefixed rows) and the job's Auto-tune activity. See Review the changes Auto-Tune applied
Output diversion planSide-by-side validation runsβ€”
Stop requestBreak test actionβ€”
Databricks API calls from inside the jobAuto-TuneYour Databricks audit log (system.access.audit), under the identity the agent calls with
Data quality metric requestsOpt-in column metrics: extra queries, or Spark Observe on running onesβ€”

The server can request only these predefined operations; it cannot send the agent code or scripts to run.

If the agent loses contact with the server, or hits an error, a timeout or excess overhead, it disables itself and the job keeps running without it. No further instructions are applied to that run.

Every agent call to the server is logged on the server side.

Supported Runtimes and Targets​

Runtime / PlatformSupportNotes
Apache Spark (Kubernetes / YARN)βœ… Full supportVersions & Details
Google Dataproc (Cluster & Serverless)βœ… Full supportIntegrated with GCP IAM and Dataproc APIs. Versions & Details
Databricksβœ… Full supportVersions & Details
Amazon EMR (Cluster & Serverless)βœ… Full supportVersions & Details
Others🧩 RoadmapAdditional runtimes under evaluation.

Security​

Security, privacy and compliance are covered in the Trust Center.