Insights & Recommendations

Definity generates insights regarding the pipelines on both Execution Health & Resource optimization:

Execution Health

Definity monitors the jobs health related aspects including failures, retries, SLA misses, disk spill, long idle time, skew patterns once a health pattern is consistent across multiple runs definity will create an insight to expose it to the owner & suggest resolution.

Resource Optimization

Spark job resource cost is usually well known and available in your cloud finOps tools. But resource waste is hard to analyze and attribute. If your job requested 50 executors with 2 vcores and 32GB RAM

1. How much does it actually use?
2. How much does can be saved without harming your SLA and how?

Definity tracks the utilization of each query & job across time, expose the waste and recommends actionable tuning.

Insights contain the following:

1. The type of insight: Cost / Health and specific information
2. Impact: $ or SLA impact value of this issue.
3. Relevant asset analyzed (the table / task)
4. Content - Detailed issue description
5. Suggested Action - definity's actionable path to resolve
6. Auto Apply option
7. Proof points - specific metrics to prove the insight & allow user deep dive