Skip to main content

Quick Start

Connect your pipelines to definity: generate an agent token, set the core configuration for your platform, then follow its setup guide.

1. Generate an agent token​

The Spark and DBT agents authenticate with an agent token. Databricks Serverless does not use one.

In the Definity app, click your user avatar (top right) and select Generate Token.

User menu — Generate Token

Select the Agent token type, optionally add a label, and click Generate. Copy the token.

Generate Agent Token dialog

Store the token in your platform's secrets manager rather than in code; each setup guide shows how.

2. Core configuration​

Spark​

At minimum, set these Spark configuration parameters:

ParameterDescriptionExample
spark.jarsURL to the Definity agent JARdefinity-spark-agent-[X.X]-latest.jar
spark.pluginsDefinity plugin (Spark 3.x)ai.definity.spark.plugin.DefinitySparkPlugin
spark.definity.serverDefinity server URLhttps://app.definity.run
spark.definity.agent.tokenAgent authentication token(your token)
spark.definity.pipeline.nameName of the pipelinemy-pipeline
spark.definity.task.nameName of the taskmy-task

The agent JAR for your Spark version is listed in Installation. All parameters are in Configuration.

DBT​

Set these environment variables, then run definity-dbt in place of dbt:

NameValueNote
DEFINITY_SERVER_URLhttps://app.definity.runmandatory
DEFINITY_TOKENyour agent tokenmandatory
DEFINITY_ENVe.g. prod, stagingoptional

Pipeline and task names default to the dbt project name. All variables are in DBT.

Databricks Serverless​

No agent, token or parameters: definity reads your workspace through a read-only service principal. See Databricks Serverless.

3. Connect your platform​

PlatformWhat is specificSetup guide
PySpark / spark-submitParameters in SparkSession.builder.config(...) or --confPySpark
Apache AirflowToken in an Airflow Variable, pipeline and task names from the DAG contextAirflow
Databricks (clusters)Init script; token in a Databricks secretDatabricks
Amazon EMR (cluster & Serverless)Bootstrap action or per job; token in AWS Secrets ManagerAmazon EMR
Google Dataproc (cluster & Serverless)Initialization action or per job; token in Google Secret ManagerGoogle Dataproc
Spark on Kubernetes or YARN, other SparkAdd the JAR and parameters to your Spark configurationInstallation
DBTpip install definity-agent-dbt, then definity-dbt in place of dbtDBT
Databricks ServerlessOne-time workspace grants, nothing installedDatabricks Serverless