Skip to main content

Prerequisites

info

Quick and Seamless Onboarding​

Start monitoring your data pipelines in under two minutes with our streamlined setup process. We support a variety of platforms, including Apache Spark and DBT. Follow these simple steps to get started:

1. Select Your Platform​

Choose the data processing platform you wish to monitor, such as Spark or DBT.

2. Generate a Secure Access Token​

Create a unique token that authorizes your environment to connect with our monitoring service.

3. Configure Required Parameters​

Define the necessary settings—such as data source paths, authentication credentials, and pipeline identifiers.

4. Deploy the Monitoring Agent​

Run the agent within your infrastructure to begin capturing real-time performance and reliability metrics.

Once deployed, you'll gain immediate visibility into your pipelines, allowing for proactive issue detection and continuous optimization.

System Requirements​

Supported Environments​

  • Spark Versions: Supports 2.3 – 4.1
  • Compute Environments:
    • Databricks
    • Databricks Serverless (jobs and SQL warehouses, through a read-only workspace connection with no agent. See Databricks Serverless)
    • Amazon EMR
    • Amazon EMR Serverless
    • Google Cloud Dataproc
    • Azure HDInsight
    • Spark on Kubernetes
    • Spark on YARN
    • Spark Connect

Network Access​

definity is a SaaS platform, so there is no server to install. Your pipelines only need to reach it:

  • Your Spark environment needs outbound HTTPS access (TCP 443) to app.definity.run. Depending on your cloud provider, this may require firewall or egress-proxy rules.
  • If your policy rules out traffic over the public internet, agents can connect over Private Link instead. See Deployment Models.
tip
  • definity primarily processes metadata required for analysis. The agent transmits SQL queries and execution plans, and samples specific data values when needed for bottleneck analysis. See what leaves your environment.
  • definity has a SOC 2 Type II report and is trusted by Fortune 500 financial services

Get Started​

Sign up here to create your account, then set up the agent for your first pipeline.

info

If you have questions or need help choosing the right setup for your team, reach out to us at [email protected].

Last updated on