Spark Agent
Definity's Spark Agent provides comprehensive observability for your Apache Spark applications, capturing lineage, metrics, and execution details automatically.
How it attaches​
The agent is a JAR added to your Spark configuration (spark.jars, plus the definity plugin spark.plugins on Spark 3.x). It requires zero code change:
- The agent automatically hooks into the Spark runtime at execution time.
- It extracts execution context, schema evolution, and metric data from runtime metadata.
- No modifications to job code, SQL, or DAGs are required.
Supported platforms​
Spark versions 2.3 – 4.1.
| Platform | Setup guide |
|---|---|
| Standalone PySpark / spark-submit | PySpark |
| Databricks (clusters) | Databricks. For serverless compute, see Databricks Serverless |
| Amazon EMR (cluster & Serverless) | Amazon EMR |
| Google Dataproc (cluster & Serverless) | Google Dataproc |
| Apache Airflow | Airflow |
| Spark on Kubernetes or YARN, Azure HDInsight, Spark Connect | Installation |
Reference​
- Installation - Agent JARs, Maven dependency and available versions
- Agent Reference - Agent footprint, configuration, tracking modes and custom metrics
New to definity? Start with the Quick Start.