Skip to main content

Architecture & Security

Overview​

Definity is an enterprise lakehouse optimization and observability platform. It connects to the enterprise data lakehouse and allows the data teams to monitor, optimize and protect their data infrastructure.

Definity can be deployed in 2 methods:

  1. A standard SaaS model hosted in the Definity's secure cloud environment. In this model metrics & metadata are reported to Definity servers.
info

Only Metadata - no data is reported to Definity

  1. A fully internal service deployment if security requirements will not allow SaaS access.
info

In internal mode no data leaves the customer's premises. Definity team has zero access to the deployment or to reported metrics

Components​

Agents​

Definity uses dedicated data pipeline agents that are added by configuration to each pipeline in the customer's data platform. The agents are provided as a code library (JAR File / Python library) and the agent code runs as part of the customer's data application. The agents communicate with Definity platform server either in the SaaS or internal deployment as selected.

The agents send metadata and metrics to the server. The server persists the metrics, runs analysis on the metrics to identify potential issues and optimizations in the user data pipelines, and exposes a web application for the pipeline owners.

Agent instrumentation​

Definity agents are added to the data pipeline and run as part of the pipeline. Specific instrumentation info can be found in the relevant installation section.

Server​

SaaS Deployment​

The Definity platform server is hosted on Definity's cloud provider. All communications with the agents and the web application are TLS encrypted and authenticated with OAuth token.

Definity extracts and transmits only statistical metrics and metadata from the data pipelines. No data is being either digested or sent to Definity.

SaaS deployment

Customer's internal Deployment​

Definity server can be deployed as a service in the customer internal network / VPC. This means no data enters or leaves the customer's premises and Definity team has no access to the deployment. This option requires a dedicated DevOps/champion to support server upgrades and maintenance.

The server deployment is provided as a standard helm / docker as described in the installation page.

Internal deployment

Zero-Code Change integration​

Definity achieves zero code change instrumentation via agent-based instrumentation:

  • The agent automatically hooks into pipeline runtimes (e.g., Spark, DBT) at execution time.

  • It extracts execution context, schema evolution, and metric data from runtime metadata.

  • No modifications to job code, SQL, or DAGs are required.

Data Flow Summary​

  1. Pipeline job starts in customer environment.

  2. Definity agent attaches at runtime and observes job metadata.

  3. Only metrics and metadata are transmitted via TLS to Definity’s control plane.

  4. Insights are stored, analyzed, and surfaced in dashboards and APIs.

Data TypeCollectedNotes
Pipeline Metadata✅Job name, start/stop time, run status, DAG/task IDs
Schema Metadata✅Column names, data types, evolution history
Metrics✅Execution metrics (duration, counts, error rates, resource usage), Column distribution statistics

No raw data, PII, or business logic leaves the customer’s environment.

Supported Runtimes and Targets​

Runtime / PlatformSupportNotes
Apache Spark (Kubernetes / YARN)✅ Full supportVersions & Details
Google Dataproc (Cluster & Serverless)✅ Full supportIntegrated with GCP IAM and Dataproc APIs. Versions & Details
**Databricks **✅ Full supportVersions & Details
Others🧩 RoadmapAdditional runtimes under evaluation.

Security, Privacy & Compliance​

Storage & Retention​

  • Telemetry Storage: Stored in Definity-managed or customer-managed databases (depending on mode).

  • Retention: Default 180 days for SaaS; configurable for private/self-hosted.

  • Encryption:

    • In Transit: TLS 1.2+

    • At Rest: AES-256

  • Backups: Automated, encrypted, and rotated.

Access Controls & Audit​

  • Authentication: OAuth 2.0 or SSO-based identity.

  • Authorization: RBAC governs all API/UI access.

  • Audit Logs: All administrative actions, API requests, and agent registrations are logged immutably.

  • Least Privilege: Each agent receives scoped tokens with limited lifetime.

Network​

  • Outbound-only HTTPS (TCP 443) required from agent to Definity server.

  • No inbound ports or firewall exceptions needed.

  • Optional customer-controlled private endpoint for Private SaaS deployments.

  • All traffic signed and verified with mutual TLS (mTLS optional in Enterprise tier).

Isolation and Fault Tolerance​

  • Agent Isolation: Agents implement multiple protections to never interfere with pipeline execution.

  • Failure Isolation: If Definity is unreachable, agents are automatically paused and continue job execution unaffected.

  • Platform Resilience: Stateless services with automatic recovery & Zero effect on customer environment.

Compliance & Governance​

AreaDefinity Practices
Security FrameworksSOC 2 Type II approved
Data PrivacyGDPR and CCPA compliant; no PII collection.
Logging & MonitoringCentralized, tamper-resistant logging; monitored 24×7.
Incident ResponseDocumented IR policy with SLA-based customer notification.
Employee AccessRole-based internal access with MFA and background checks.
Last updated on