Skip to main content

Custom Metrics

Definity allows you to report custom user-defined metrics from your Spark applications. These metrics are automatically captured and displayed in Definity UI, enabling you to track KPIs, data quality checks, and other domain-specific measurements alongside Definity's data and execution metrics.

How It Works​

To report custom metrics, structure your query to return two specific columns:

Column NameTypeDescription
definity_metric_nameStringThe name/identifier for your metric
definity_metric_valueNumericThe numeric value of the metric

Basic Example​

spark.sql(
  """
    SELECT
        'transactions.revenue' as definity_metric_name,
        SUM(amount)            as definity_metric_value
    FROM transactions
  """
).collect()

Advanced Example: Multiple Metrics​

You can report multiple metrics in a single query by generating multiple rows:

spark.sql(
  """
    SELECT
        CONCAT('revenue.market.', market_id) as definity_metric_name,
        AVG(amount)                          as definity_metric_value
    FROM transactions
    GROUP BY market_id

    UNION ALL

    SELECT
        'transactions.count' as definity_metric_name,
        COUNT(*)             as definity_metric_value
    FROM transactions
  """
).collect()

Best Practices​

  • Naming Convention: Use descriptive, hierarchical names (e.g., revenue.daily.total, quality.user_id.null_rate) as this is how they will appear in the UI
  • Metric Cardinality: Be mindful of the number of metrics generated, especially when using dynamic names with high-cardinality dimensions
Last updated on