DataProc

Run on Google cloud's dataproc:

gcloud dataproc batches submit \
    --project your-project \
    --region us-central1 \
    spark \
    --version 1.2 \
    --subnet default \
    --class com.pany.spark.SomeSparkJob \
    --jars gs://your-bucket/definity-spark-agent-X-X.jar \
    --properties spark.extraListeners=ai.definity.spark.AppListener,spark.definity.server=https://app.definity.run,spark.definity.api.token=$DEFINITY_API_TOKEN,spark.definity.env.name=demo,spark.definity.pipeline.name=example_pipeline