Skip to main content

Agent Footprint

Definity agents are designed to have negligible impact on your production data pipelines. The agents run passively within your existing Spark and DBT jobs, collecting observability data with minimal resource overhead.

Types of Instrumentation​

Definity's agent uses three types of instrumentation, each with a different footprint profile:

1. Driver Metadata Collection​

Metrics collected in the Spark driver or DBT process by hooking into the native event system - no additional computation required:

  • Lineage - Captured from Spark's query plans
  • Row counts, bytes, partitions - From Spark's scan/write metrics
  • Execution times, stage and task counts - From Spark's event listeners
  • Schema information - From catalog metadata
  • DBT model results - From DBT adapter statistics
  • DBT scripted models (BigQuery) - Lineage and per-statement details for models built by custom materializations are recovered from the warehouse's own job metadata (Jobs API), plus catalog lookups for input freshness/volume (a single batched metadata query per model) — metadata-only calls, no data scans

Footprint: Purely passive - metrics are a byproduct of normal execution; the BigQuery metadata lookups are catalog/API reads that trigger no additional data processing.

2. Executor Observability (Spark only)​

A Java agent runs inside each executor JVM, enabling deeper observability beyond what the driver can see:

  • Skew key analysis - Identifying skewed keys by sampling data during reads/writes
  • Thread dumps - Capturing executor thread state to diagnose performance issues
  • Executor and node info - Machine-level metrics such as CPU, memory, and executor environment details

Footprint: The Java agent intercepts data and JVM state in-process with negligible overhead through sampling. No additional data scans are triggered.

3. Data Quality Queries (Opt-In)​

When users explicitly enable data quality metrics for specific columns, Definity either adds dedicated queries or extends already-running queries using Spark Observe to collect the requested metrics in a single pass:

  • Null percentage - Count aggregation per column
  • Distinct values - Distinct count per column
  • Value distributions - Histogram of column values

Footprint: Controlled and explicit - only runs for columns the customer has opted into, with Spark Observe minimizing the overhead by piggybacking on existing data scans where possible.

Measured Resource Impact​

Definity continuously monitors its own agent footprint through dedicated metrics (definity_driver_overhead and definity_executors_overhead). Based on over 1.5 million production task executions:

Driver Overhead​

Based on our measurements and analysis, driver overhead is approximately ~0.5% for short-running tasks (up to 5 minutes), and lower as tasks get longer.

Executor Overhead​

Executor overhead is measured as accumulated vcore-time across all executors. The cost impact is < 0.2% of total compute resources.

Real Production Data

These measurements are derived from ~1M task executions across production Spark workloads.

Customer Data Collection​

Definity agents collect metadata about the data, not the data itself:

  • ✅ Table names, column names, schemas
  • ✅ Row counts, null counts, distinct counts
  • ✅ Execution times, resource usage
  • ⚠️ A few row values for diagnosis – the most frequent keys in the slowest tasks and the row a failed task was on
  • ⚠️ Query text and plans, including literals
  • ⚠️ Column value histograms, only for columns you opt in
  • ❌ Query results or bulk data

Set spark.definity.privacy.enabled to stop sending any value read from your rows.

How Definity Minimizes Footprint​

Definity is designed from the ground up to minimize impact on production workloads:

Passive Observation Architecture​

  1. Leverage Spark's existing metadata - Extract metrics from Spark's query execution listeners and metrics that are already computed
  2. No additional queries by default - Metadata metrics require zero extra data scans or queries
  3. Event-driven collection - Hook into Spark's native event system rather than actively polling

Efficient Execution Strategy​

  1. Asynchronous reporting - Metrics are reported to the server asynchronously, never blocking job completion
  2. Batched updates - Multiple metrics are batched together to minimize network calls
  3. Selective computation - Distribution metrics (requiring extra queries) are opt-in only

Summary​

Definity's agent is designed to be a transparent observer of your data pipelines, backed by real production data:

  • Driver overhead: ~0.5% or less of task runtime (based on production analysis)
  • Executor overhead: < 0.2% of vcore-time/compute cost (negligible resource impact)
  • No additional data scans: Metadata collection uses existing Spark execution data
  • Opt-in data quality metrics: Only run when explicitly configured per column

Observing is not all the agent does: it also applies autotuning and configuration overrides, adds data quality queries, diverts output for shadow runs, and can stop a run. Set spark.definity.readOnly to keep it to observing and reporting.

Bottom line: Definity provides comprehensive observability (lineage, metrics, execution details) by passively observing Spark's existing query execution - no additional data scans or queries for metadata instrumentation, and negligible compute cost for executor-side collection.

Last updated on