Google DataProc

Compatibility Matrix

Dataproc Image	Spark Version	Scala Version	Definity Agent
2.3	3.5.3	2.12.18	3.5_2.12-latest
2.2	3.5.3	2.12.18	3.5_2.12-latest
2.1	3.3.2	2.12.18	3.3_2.12-latest
2.0	3.1.3	2.12.14	3.1_2.12-latest
1.5	2.4.8	2.12.10	2.4_2.12-latest
1.4	2.4.8	2.11.12	2.4_2.11-latest
1.3	2.3.4	2.11.8	2.3_2.11-latest

Job Submission Example

Run on Google cloud's dataproc:

gcloud dataproc batches submit \
    --project your-project \
    --region us-central1 \
    spark \
    --version 1.2 \
    --subnet default \
    --class com.pany.spark.SomeSparkJob \
    --jars gs://your-bucket/definity-spark-agent-X-X.jar \
    --properties spark.plugins=ai.definity.spark.plugin.DefinitySparkPlugin,spark.definity.server=https://app.definity.run,spark.definity.api.token=$DEFINITY_API_TOKEN,spark.definity.env.name=demo,spark.definity.pipeline.name=example_pipeline