Skip to main content

Spark Agent

Definity's Spark Agent provides comprehensive observability for your Apache Spark applications, capturing lineage, metrics, and execution details automatically.

How it attaches​

The agent is a JAR added to your Spark configuration (spark.jars, plus the definity plugin spark.plugins on Spark 3.x). It requires zero code change:

  • The agent automatically hooks into the Spark runtime at execution time.
  • It extracts execution context, schema evolution, and metric data from runtime metadata.
  • No modifications to job code, SQL, or DAGs are required.

Supported platforms​

Spark versions 2.3 – 4.1.

PlatformSetup guide
Standalone PySpark / spark-submitPySpark
Databricks (clusters)Databricks. For serverless compute, see Databricks Serverless
Amazon EMR (cluster & Serverless)Amazon EMR
Google Dataproc (cluster & Serverless)Google Dataproc
Apache AirflowAirflow
Spark on Kubernetes or YARN, Azure HDInsight, Spark ConnectInstallation

Reference​

  • Installation - Agent JARs, Maven dependency and available versions
  • Agent Reference - Agent footprint, configuration, tracking modes and custom metrics

New to definity? Start with the Quick Start.