What is definity
definity optimizes and protects your data lakehouse for you​
definity is built for the data lakehouse, whichever flavor you run: Databricks, Amazon EMR, Google Dataproc, Azure HDInsight, open-source Spark on Kubernetes or YARN, and dbt on warehouses such as BigQuery and Snowflake. Its agents run inside your production pipelines and act on what they find. They tune each job on its real runs, check every change against the job's SLA, duration and reliability, and revert any change that causes a regression.
- Automated optimization: Auto-Tune adjusts configuration such as executors, cores, memory and shuffle partitions one step at a time, keeps what holds, and re-tunes when a job drifts. Where a fix needs a code change, definity gives engineers the exact change and its annual impact.
- Self-protecting pipelines: tests are generated automatically from each pipeline's history. When one fails, the agent can stop the run or divert its output before bad data reaches downstream.
- Automated root-cause analysis: definity correlates code, configuration, environment, schema and data changes to explain why a run failed, slowed down or produced bad data.
- Automated change validation: definity re-runs your jobs side by side on full-scale data to validate version and infrastructure upgrades before they reach production.
This is done centrally across the entire lakehouse, with zero code changes.
Solution​
Data Platform Teams​
Automated resource optimization​
definity finds over-provisioning, data skew and missing parallelization in every pipeline, and Auto-Tune fixes the configuration waste on production runs, without SLA impact.
Impact: lower compute cost, better performance and fewer SLA misses across the whole lakehouse, with no tuning work for engineers.
Automated reliability​
definity catches failures, retries and runs at risk of missing their SLA as they happen, and explains each one with automated root-cause analysis.
Impact: less downtime, and far less engineering time spent on investigations.
Central, automatic coverage​
definity is added once, centrally, on each lakehouse platform you run, and covers every pipeline in the organization, existing and new, with no code changes.
Impact: optimization and protection reach the whole lakehouse, not only the pipelines someone had time to configure.
Automated change validation​
definity re-runs jobs side by side on full-scale data to validate upgrades, migrations and platform changes, and stops breaking schema changes before they reach downstream.
Impact: upgrades and migrations ship without production incidents.
To get started create your account and add pipelines with our free tier now
Data Engineering Teams​
Auto-generated tests and data contracts​
definity learns each pipeline's normal behavior and generates tests and data contracts for freshness, volume, completeness, schema and distribution. They update as the pipeline changes, with no rules to write.
Impact: high test coverage from day one, and issues caught as they happen.
Real-time protection​
The agent checks data while the pipeline runs. On stale or faulty input it can stop the run, and on faulty output it can divert the data to a quarantine location, so bad data never reaches downstream.
Impact: incidents are stopped before they reach the business.
Automated root-cause analysis​
definity puts code changes, environment changes, schema changes and data shifts in context, and traces issues across full-stack lineage, including intermediate tables, to pinpoint what caused an incident.
Impact: shorter time to resolve, with less engineering and compute spent on debugging.
Learn how automated tests are created, and how full lineage traces incidents to their root cause.