Tests
A test is defined as the valid range of values for a specific metric. If in a new execution of the pipeline the metric will have a value outside this range, the test will fail and an alert will be generated.
definity Test Engine will only start to generate automated tests for a specific metric (for a specific pipeline) after a baseline for the metric was established.
- I.e., tests are only created after ~4-5 pipeline executions are completed successfully with the specific metric.
- If a new metric is opted-in, ~4-5 successful pipeline executions are also required (to establish a baseline), before tests will be generated for it.
Full tests list & details
| Test Type | Validation | Parameters | Example |
|---|---|---|---|
| Const | Test metrics than must retain 1 single value, for example schema_cnt | Single value | 18 |
| Pct Diff | Test metric value change compared to the average of the last N values of this metric | 2 values comma separated: allowed percent delta, window size | 12, 3 |
| Range | A constant max and min value | 2 values comma separated: min,max | 2000, 2700 |
| Trend | A linear regression test - metrics must be within the tolerance distance from the linear functions | 3 values comma separated: slope, y, tolerance | -0.000034, -3.7,0.49 |
Tests configuration
- Review the automatic tests created by definity Test Engine
- Configure tests manually - click the edit pen and select the test type, values and action
Test action definition:
| Name | Action | Comments |
|---|---|---|
| Alert | Raise an alert | Currently alerts are internal in Definity. Email/Slack integration will be added |
| Pass | Ignore test failure | Used for temporary muting tests |
| Break | Stop pipeline execution | Currently not supported |