Core Concepts

Agents that act, not dashboards

definity's agents run inside your lakehouse pipelines, on Databricks, EMR, Dataproc, Spark on Kubernetes or YARN, and dbt. They don't stop at showing you what's wrong. Each agent works in a closed loop:

1. Observe: collect execution, resource and data metrics from every run, as it happens.
2. Decide: learn each pipeline's normal behavior and find waste, risk and bad data.
3. Act: change the job's configuration, stop a run, or divert its output.
4. Verify: check the result on the next runs, and revert anything that made the job worse.

Your teams keep their focus on business logic. Tuning and protecting pipelines is handled for them.

Auto-Tune: closed-loop optimization

Auto-Tune is the closed loop applied to cost and performance. definity finds the waste in each job, such as over-provisioned executors, memory or shuffle partitions, and fixes it on real production runs:

• It records a baseline, applies one change, and measures the result.
• It keeps a change only if the run stays within the job's duration budget, meets its SLA, stays reliable and costs less. Otherwise it reverts the change and tries a safer path.
• It settles on the best safe configuration and keeps watching. If the job drifts, for example because input volumes grow, it tunes again.

Auto-Tune is enabled across the workspace and needs no code changes. Waste that can only be fixed in code, such as a skewed key or a query rewrite, comes as a specific recommendation with its annual impact.

Learned behavior, no rules to write

definity learns each pipeline's baseline from its own history across hundreds of metrics, covering execution health, data health and resource utilization. From that baseline it generates, on its own:

1. Data quality tests and data contracts
2. Execution health tests
3. Health insights
4. Optimizations, applied by Auto-Tune where they can be

Tests and optimizations update as pipelines change, so coverage doesn't depend on anyone writing or maintaining rules.

Real-time intervention

Traditional data quality tools check the data after the pipeline has ended. definity's agent runs as part of the pipeline itself, with negligible footprint, so it can act before an incident spreads:

Real-time analysis	Automatic intervention	Impact
Identify stale input data before the application starts to transform it	Stop the pipeline before it runs	Resources saved, and no downstream impact from stale data
Identify stuck jobs early	Stop the stuck jobs automatically	Meet SLA, allow a rerun, and save wasted resources
Identify faulty output data at runtime	Divert the final output to an alternate location	No downstream contamination, while allowing debugging and approval

Bounded by design

Agents act only through a fixed set of predefined operations: configuration overrides, compute provider API calls for Auto-Tune, output diversion, stopping a run, and extra data quality queries. The server can't send the agent new code. Any change that makes a job worse is reverted, and if the agent hits an error or loses contact with the server, it disables itself and the job runs on unaffected.

See Architecture for the full list of operations.

Full-stack context

The agents act on the full stack, not only the data. They see the code and configuration that produce and consume the data, the platform it runs on, and the data itself:

1. Application health and stability
2. Performance, resource efficiency and cost
3. The quality of the processed data, tested for the needs of each application that reads it

That context is what lets definity tune a job safely, and trace an incident to its root cause across code, environment, schema and data changes.