Quick Start
Connect your pipelines to definity: generate an agent token, set the core configuration for your platform, then follow its setup guide.
1. Generate an agent token​
The Spark and DBT agents authenticate with an agent token. Databricks Serverless does not use one.
In the Definity app, click your user avatar (top right) and select Generate Token.

Select the Agent token type, optionally add a label, and click Generate. Copy the token.

Store the token in your platform's secrets manager rather than in code; each setup guide shows how.
2. Core configuration​
Spark​
At minimum, set these Spark configuration parameters:
| Parameter | Description | Example |
|---|---|---|
spark.jars | URL to the Definity agent JAR | definity-spark-agent-[X.X]-latest.jar |
spark.plugins | Definity plugin (Spark 3.x) | ai.definity.spark.plugin.DefinitySparkPlugin |
spark.definity.server | Definity server URL | https://app.definity.run |
spark.definity.agent.token | Agent authentication token | (your token) |
spark.definity.pipeline.name | Name of the pipeline | my-pipeline |
spark.definity.task.name | Name of the task | my-task |
The agent JAR for your Spark version is listed in Installation. All parameters are in Configuration.
DBT​
Set these environment variables, then run definity-dbt in place of dbt:
| Name | Value | Note |
|---|---|---|
DEFINITY_SERVER_URL | https://app.definity.run | mandatory |
DEFINITY_TOKEN | your agent token | mandatory |
DEFINITY_ENV | e.g. prod, staging | optional |
Pipeline and task names default to the dbt project name. All variables are in DBT.
Databricks Serverless​
No agent, token or parameters: definity reads your workspace through a read-only service principal. See Databricks Serverless.
3. Connect your platform​
| Platform | What is specific | Setup guide |
|---|---|---|
| PySpark / spark-submit | Parameters in SparkSession.builder.config(...) or --conf | PySpark |
| Apache Airflow | Token in an Airflow Variable, pipeline and task names from the DAG context | Airflow |
| Databricks (clusters) | Init script; token in a Databricks secret | Databricks |
| Amazon EMR (cluster & Serverless) | Bootstrap action or per job; token in AWS Secrets Manager | Amazon EMR |
| Google Dataproc (cluster & Serverless) | Initialization action or per job; token in Google Secret Manager | Google Dataproc |
| Spark on Kubernetes or YARN, other Spark | Add the JAR and parameters to your Spark configuration | Installation |
| DBT | pip install definity-agent-dbt, then definity-dbt in place of dbt | DBT |
| Databricks Serverless | One-time workspace grants, nothing installed | Databricks Serverless |