Skip to main content

Performance Overview

info

definity removes resource waste from your Spark pipelines automatically. It finds where each job wastes resources, and Auto-Tune fixes the waste that configuration can fix, directly on production runs and without harming the job's SLA. Waste that needs a code change comes with a specific recommendation.

  1. Find the waste in every pipeline, with its annual impact, so the highest-value fixes come first.
  2. Fix it automatically with Auto-Tune: right-size and re-tune each job's configuration, with every change validated and regressions reverted.
  3. Recommend code fixes for the waste configuration can't solve, with their projected impact.

Resource Waste in Spark Pipelines​

definity categorizes resource waste in Spark pipelines into two main types:

1. Unutilized Resources​

By following each pipeline's behavior over time, definity determines the resources it actually needs, filtering out noise from short-term peaks and lows. It identifies:

  • vCores that are provisioned but never used.
  • Memory that is allocated but remains unused.
  • vCores that appear to be utilized but remain idle during execution.

2. Inefficient Resource Utilization​

Resource inefficiency occurs when the allocated resources are used in a suboptimal manner. definity identifies and analyzes factors such as:

  • Failures & Retries – Frequent task failures leading to unnecessary resource consumption.
  • Skewed Workloads – Data imbalances causing uneven task execution across nodes.
  • High Shuffle & Spill Costs – Excessive disk and network I/O from suboptimal Spark operations.
  • Suboptimal Partitioning – Inefficient partitioning leading to performance bottlenecks.
  • Task Parallelism Issues – Misconfigured concurrency settings resulting in underutilized cluster capacity.

How the Waste Gets Fixed​

Automatically, with Auto-Tune​

Waste that can be fixed through configuration is tuned by definity on the job's real production runs:

  • Right-sizing executor count, cores and memory to what the job actually uses.
  • Parallelism, such as the number of shuffle partitions, to match the job's data.

definity applies one change at a time and keeps it only if the run stays within the job's duration budget, meets its SLA, stays reliable and costs less. Anything that regresses is reverted, and the job is re-tuned when its workload drifts. Turn on Auto-Tune once, for the whole workspace, and every job is covered. See Auto-Tune.

By your team, with a recommendation​

Waste that needs a code or data change comes as a specific recommendation with its projected impact:

  • Data partitioning and skewed keys in the job's code.
  • Code and query optimization, such as rewriting inefficient queries and Spark transformations.
tip

The result: lower infrastructure cost, better performance and fewer SLA misses across your Spark pipelines, with no tuning work for your engineers.