Example: Over-Provisioning

Definity's dashboard provides insights into pipelines with excessive resource waste, highlighting opportunities for cost savings.

Identifying Waste in Pipelines

The dashboard allows users to analyze specific pipelines and categorize different types of waste.

In this example, the waste is fully attributed to memory over-provisioning.

Analyzing Cost Impact & Optimization Suggestions

By clicking on the lamp icon, users can access detailed insights, revealing an annual cost impact of $48K due to inefficient memory allocation.

Definity recommends setting spark.executor.memory to 31, based on a thorough review of historical execution data.

Automated Optimization & Future Execution Improvements

Definity can automatically apply this optimization to future executions without requiring code changes, provided the pipeline owner approves.

Drilling down further into execution data, we see a consistent pattern of unused memory, reinforcing the potential savings from right-sizing resource allocation.

By leveraging these insights, users can reduce costs, improve efficiency, and ensure optimal resource utilization across their pipelines.