Configuration Reference

Core Configuration Parameters

Name	Details
`spark.jars`	URL of `definity-spark-agent-X.X.jar` (and optionally `definity-spark-iceberg-1.2-X.X.jar`)
`spark.plugins`	Add `ai.definity.spark.plugin.DefinitySparkPlugin` (for Spark 3.x)
`spark.extraListeners`	Add `ai.definity.spark.AppListener` (for Spark 2.x)
`spark.executor.plugins`	Add `ai.definity.spark.plugin.executor.DefinityExecutorPlugin` (for Spark 2.x)
`spark.definity.server`	Definity server URL (e.g., `https://app.definity.run`)
`spark.definity.agent.token`	Agent token (required for SaaS usage). Can also read from driver's env var `DEFINITY_AGENT_TOKEN`.

Pipeline Tracking Parameters

These parameters enable tracking and monitoring of your Spark application's execution over time. Consistent naming allows you to correlate metrics and logs across multiple runs.

To group multiple tasks into the same pipeline run, use either pipeline.pit or pipeline.run.id (but not both - they are mutually exclusive):

pipeline.pit (recommended): Groups tasks by a shared logical point-in-time, which is also used as each task's app_pit.
pipeline.run.id: Groups tasks by an arbitrary identifier. When using this, each task's app_pit defaults to run time of the first task that share the same pipeline.run.id.

Name	Details
`spark.definity.env.name`	defaults to `default`
`spark.definity.pipeline.name`	defaults to `spark.app.name`
`spark.definity.pipeline.pit`	the logical Point-in-Time of a run; defaults to `now` (see supported formats below)
`spark.definity.pipeline.run.id`	alternative to `pit` for grouping tasks; use when a logical time isn't available
`spark.definity.task.name`	defaults to `spark.app.name`

Supported PIT Formats

The pipeline.pit parameter accepts the following date/time formats:

Format	Example
`YYYY-MM-DD HH:MM:SS`	`2020-05-03 13:15:00`
`YYYY-MM-DDTHH:MM:SS`	`2020-05-04T13:15:00`
`YYYY-MM-DD HH`	`2020-05-03 12`
`YYYY/MM/DD HH:MM`	`2020/05/03 14:10`
`YYYY_MM_DD`	`2020_05_03`
`YYYY-MM-DD_HH`	`2020-05-03_12`
Unix timestamp (seconds)	`1688289300`
Unix timestamp (milliseconds)	`1688289300000`
ISO 8601 with microseconds	`2023-07-18T19:17:41.286948`
ISO 8601 with timezone	`2023-07-18T19:17:41+00:00`
ISO 8601 with Z suffix	`2025-03-13T14:00:00Z`

Streaming

Name	Details
`spark.definity.streamingSession.rotation.enabled`	Enable session rotation for infinite streaming apps; defaults to `true`.
`spark.definity.session.rotation.seconds`	Maximum duration in seconds for sessions before rotation; defaults to `43200` (12 hours).

Advanced Configuration

Name	Details
`spark.definity.enabled`	Enables or disables functionality with options: `true`, `false`, or `opt-in` (default: `true`). For `opt-in`, users can toggle this in the pipeline settings page.
`spark.definity.task.id`	User-defined task ID to show in the UI and notifications (e.g., YARN run ID); defaults to `spark.app.name`.
`spark.definity.tags`	Comma-separated tags, supports `key:value` format (e.g., `team:team-A`).
`spark.definity.email.to`	Comma-separated list of notification recipient emails.
`spark.definity.task.heartbeat.interval`	Interval in seconds for sending heartbeat to the server; defaults to `120`.
`spark.definity.server.request.retry.count`	Number of retries for server request errors; defaults to `3`.
`spark.definity.ignoredTables`	Comma-separated list of tables to ignore. Names can be full (e.g., `db_a.table_a`) or partial (e.g., `table_a`), which applies to all databases.
`spark.definity.files.sanitizedNamePattern`	Regular expression to extract time partitions from file names. Defaults to `^.*?(?=/\d+/\|/[^/]_=[^/]_/)`. Set empty to disable.
`spark.definity.inputs.maxPerQuery`	Maximum number of allowed inputs per query; defaults to `100`.
`spark.definity.default.session.enabled`	Enables default session for multi-concurrent SparkSession apps; defaults to `true`. Set to `false` to disable.
`spark.definity.metrics.injection.enabled`	Enable in flight data distribution metrics; defaults to `false`.
`spark.definity.debug`	Enable debug logs; defaults to `false`.
`spark.definity.databricks.automaticSessions.enabled`	Enable auto detection of tasks in Databricks multi-task workflows; defaults to `false`. defaults to `true`.
`spark.definity.events.enabled`	Flag to enable reporting of events. defaults to `true`.
`spark.definity.events.maxPerTaskRun`	Maximum number of events to report in one task. defaults to `1000`.
`spark.definity.plugin.executor.enabled`	Enables executor side plugin when definity plugin is configured; defaults to `true`.

Metrics Calculation

Name	Details
`spark.definity.num.threads`	Number of threads for metrics calculation; defaults to `2`.
`spark.definity.metrics.timeout`	Timeout for metrics calculation, in seconds; defaults to `180`.
`spark.definity.metrics.histogram.maxNumValues`	Maximum number of values for histogram distribution; defaults to `10`.
`spark.definity.metrics.timeSeries.initialBucketSizeSeconds`	Time-series metrics initial bucket size in seconds; defaults to `30`.
`spark.definity.driver.containerMemory`	Total container memory for the driver in bytes (for client mode).
`spark.definity.driver.heapMemory`	Total heap memory for the driver in bytes (for client mode).

Output Diversion (Testing & CI)

Redirects a run's outputs so it can execute against production inputs without writing to production outputs — the basis of CI shadow runs.

spark.definity.diversion.enabled is the master switch and defaults to false. With it off, the keys below are read and ignored: the run succeeds and writes to its original outputs.

Name	Details
`spark.definity.diversion.enabled`	Turns output diversion on. Defaults to `false`; without it none of the keys below have any effect.
`spark.definity.diversion.baseLocation`	Base location for everything the diverted run writes — files, databases, tables and stream checkpoints. Either a full base location, to divert everything to a single place, or a partial path to keep each output in its own bucket under a different base directory. e.g. `gs://my-tests-bucket` or `my-tests-base-dir`.
`spark.definity.diversion.database.suffix`	Suffix added to the database name of every output table.
`spark.definity.diversion.table.suffix`	Suffix added to the name of every output table.
`spark.definity.diversion.catalog`	A fixed catalog to divert all output tables into. Unlike the suffixes, it replaces the slot outright rather than deriving the target from the original name — so it needs no suffix alongside it. The catalog must already exist.
`spark.definity.diversion.bigquery.project`	BigQuery project id override for all BigQuery output tables.
`spark.definity.diversion.bigquery.dataset`	BigQuery dataset name override for all BigQuery output tables.

Set at least one target (a suffix, a fixed catalog, or the BigQuery overrides) and a base location. If a run writes somewhere the configured keys cannot cover, the agent breaks the run rather than letting the write reach the original output, and the error names the key to set.

Upgrading from before diversion.*? These keys were spark.definity.output.* in earlier agents. The old names are no longer recognised — a run still setting them writes to its original outputs — so rename them and add diversion.enabled=true.

Iceberg and Delta targets

Iceberg and Delta can divert in place — into a branch of the source table, or a clone of it — instead of into a renamed table. Row-level DML (MERGE, UPDATE, DELETE) needs this: a renamed target starts empty, so the DML only ever inserts and the result does not match what the source run produced.

Name	Details
`spark.definity.diversion.iceberg.mode`	How iceberg output tables are diverted. `table` (default) writes to a renamed table, per the suffix and catalog keys above. `branch` writes to a branch of the source table, leaving its main branch untouched. `wapBranch` uses Iceberg's write-audit-publish flow, setting `spark.wap.branch` and the table's `write.wap.enabled` property.
`spark.definity.diversion.iceberg.branch`	Branch name used by the branch-based modes. Defaults to `definity_staging`.
`spark.definity.diversion.iceberg.branch.retainDays`	How long the diverted branch is retained.
`spark.definity.diversion.delta.clone`	How the diverted Delta table is seeded from the source. `shallow` clones the source's data files by reference, so the DML sees the original rows; `empty` starts from an empty table.

Both branch modes need Iceberg ≥ 1.2 and Spark ≥ 3.3, and delta.clone=shallow needs OSS Delta ≥ 2.3 or DBR ≥ 13.3. Below those the agent logs a warning and falls back — to rename-based diversion for iceberg, and to an empty clone for Delta. Check the driver log for that warning rather than assuming the mode you asked for is the one in effect.

Skew Detection Events

Skew events are calculated in the executors and use Spark's plugins mechanism.

Name	Details
`spark.definity.plugin.executor.driverPollingIntervalSeconds`	Interval in seconds between consecutive polling requests from executor to driver when using the Definity plugin; defaults to `20`.
`spark.definity.skewDetection.minTaskSkewTimeSeconds`	Minimum difference in seconds between suspected skewed task duration and the average task duration in its stage; defaults to `60`.
`spark.definity.skewDetection.minTaskSkewFactor`	Minimum ratio between suspected skewed task duration and the average task duration in its stage; defaults to `5`.
`spark.definity.skewDetection.samplingRatio`	Sampling ratio of task rows (e.g., `0.01` equals 1% sampling); defaults to `0.01`.
`spark.definity.skewDetection.maxSampledRowsPerTask`	Maximum number of sampled rows per task; defaults to `1000`.
`spark.definity.skewDetection.maxReportedKeysPerTask`	Maximum number of reported keys per task; defaults to `3`.

Core Configuration Parameters​

Pipeline Tracking Parameters​

Supported PIT Formats​

Streaming​

Advanced Configuration​

Metrics Calculation​

Output Diversion (Testing & CI)​

Iceberg and Delta targets​

Skew Detection Events​