Skip to main content

Server Changelog

Main functional features:

[0.42.2] - 2025-04-08​

  • Dashboard zoom changes the time frame
  • Add active tasks metric
  • Add generic SSO support

[0.41.5] - 2025-03-25​

  • Fix for opt-in new pipelines
  • some UI fixes

[0.41.2] - 2025-03-14​

  • Auto tests generation improvements
  • A few bug fixes

[0.40.0] - 2025-03-09​

  • Added new on-boarding and sign-up flows
  • Added insights
    • many small files
    • skewed tasks
    • executors memory under-provisioning

[0.30.0] - 2025-02-23​

  • Added insights for idle/spill/GC time and driver unused heap
  • Health & cost dashboards improvements
  • Lineage improvements - unify data and execution, add task drill down

[0.20.3] - 2025-02-17​

  • hotfix for datasets with no schema

[0.20.2] - 2025-02-13​

  • Multi tenant issues hotfix

[0.20.1] - 2025-02-11​

  • Pipeline run notifications hotfix

[0.20.0] - 2025-01-29​

  • Major refactor of agent-server protocol (server is backward compatible)
  • Add support for pipeline-users permissions
  • Overall UI enhancements - layout, menus, etc.
  • Added time frame picker
  • Added run status of "Completed with retries"
  • Improved health dashboard
  • Improved timeline chart
  • Lineage improvements - support hopping to upstream pipelines
  • Fix wrong incidents for updating metrics

[0.15.0] - 2024-12-25​

  • Added pipeline and task insights and recommendations
  • Added events timeline to spark overview page
  • Improved health dashboard
  • Added simple retention policy
  • Added support for multi tenancy
  • Added SSO login
  • Added system setting to disable all agents

[0.14.7] - 2024-12-18​

  • Added parameters to ingress Helm chart

[0.14.6] - 2024-11-21​

  • Added SLA monthly digest email

[0.14.5] - 2024-11-21​

  • Added lineage view in compare-runs
  • Added code tab in compare-runs
  • Overall UI enhancements
  • Added cross task view in Spark-overview page
  • Fixed subscribed metrics report

[0.14.4] - 2024-11-06​

  • Enhancements to daily incidents mail
  • Add Code tab in compare-runs page
  • Add multi-thresholds in compare-runs page

[0.14.3] - 2024-11-02​

  • Add Freshness incidents to daily mail

[0.14.1] - 2024-10-27​

  • Compare-runs page enhancements
  • Fix daily SLA mail issue

[0.14.0] - 2024-10-10​

  • Added Comparative analysis page
  • Added SLA daily digest email

[0.13.0] - 2024-09-24​

  • Added incident entity to consolidate metric alerts
  • Added task idle time metric
  • Improved pipelines page

[0.12.0] - 2024-09-11​

  • Added pipelines coverage dashboard
  • Add more resource metrics to task and overtime - input/output/shuffle bytes, failures, retries, skew, broadcasts, gc, spill
  • Add support for run state management for agents

[0.11.2] - 2024-09-08​

  • Hotfix for a few edge cases

[0.11.0] - 2024-08-26​

  • Added Prophet based anomaly detection tests generator and dedicated pod/worker
  • Added Spark Driver memory timeseries metric
  • Added bulk actions option to metrics table
  • Added partition details the schema comparison page
  • Fix Spark metrics live updates
  • Enhanced the tags filter menu

[0.10.0] - 2024-07-31​

  • Added Pipeline resources and cost dashboards
  • Added Spark task resources overview page

[0.9.3] - 2024-07-08​

  • Fix handling sudden log spike

[0.9.2] - 2024-07-02​

  • Added new memory metrics - allocated, on-heap, off-heap, used, utilized for driver and executors
  • Email notification redesign
  • Added "reported metrics" section in run completed email notification

[0.9.1] - 2024-06-23​

  • Add support for files as pipeline inputs (with metrics)
  • Add "0"s in Histogram metric for values that were not reported
  • Overall many UI improvements
  • Improve metric values format depending on units
  • Improve alerts and dataset metrics pages
  • Optimize backend routes to handle higher load
  • Separate queries to another table to ease on DB export
  • Fix task duration bug

[0.8.4] - 2024-06-06​

  • Fix live data freshness metric issue
  • Fix adding missing values to histogram metric

[0.8.1] - 2024-06-05​

  • Hotfix slow heartbeat endpoint

[0.8.0] - 2024-06-02​

  • Added support for killing long running tasks ("break" on duration test)
  • Added support for notification per test (useful for Live Table Freshness metric)
  • Added user defined "Pipeline run ID" and "Task ID" identifiers
  • Unified metrics definitions across the system
  • Increased Histogram metric to 100 values
  • Improve auto tests

[0.7.6] - 2024-05-20​

  • Stability fixes
  • Enhance auto tests
  • Add live freshness metric for output tables
  • Fix alerts on "unknown" tasks
  • Improve UI responsiveness

[0.7.2] - 2024-04-19​

  • Add support for pipeline run id

[0.7.0] - 2024-04-15​

  • Add support for task-level metrics config
  • Add asset profiles to metric config
  • Add option to disable agent from UI
  • Improve email notification details
  • Improve metric names and detailed info
  • Improve auto tags logic

[0.6.4] - 2024-03-31​

  • Added support for metrics group view and tests
  • Added notification option for pipeline failed (including task retries)
  • Added metric count NaN values for numeric columns
  • Added lineage navigation in asset tree
  • Fixed missing metrics for input tables
  • Fixes for auto lineage logic
  • Fixes for auto generated tests
  • Api query optimizations

[0.6.3] - 2024-03-13​

  • Added suggested tests feature to replace "outdated" tests
  • Added notification for pipeline completed with alerts
  • Expand metrics asset tree component to all pages
  • Enable PIT navigation on the metric chart

[0.6.2] - 2024-03-07​

  • Added tests and alerts to metrics asset tree
  • Added notifications for task failed, task is hanging, pipeline completed, pipeline did not start
  • Show Spark's TF id in task page
  • Added system settings
  • Added pipeline settings
  • Added option to regenerate auto-tests per pipeline
  • Enhanced metric table - sorting, filtering, etc.
  • Added pipeline run metrics - count of tasks, cpu_time, used_cpu_time, process_time, duration, SLA
  • Fixed input table freshness metric

[0.5.2] - 2024-01-25​

  • Added Metrics - num TFs metric in task, num schema changes, task SLA
  • UI Improvements - Asset tree metrics component
  • Improved auto-tags logic
  • Added error tab in TF page

[0.5.0] - 2024-01-09​

  • Task params comparison
  • Dataset schema comparison
  • Added dedicated test drawer
  • Page titles
  • Dataset downstream tab
  • Added auto pipeline tags logics
  • Docs link in UI
  • Metric chart improvements
  • Added server version to task params
  • Cleaning Spark's query plan from temporary column numbers

[0.4.2] - 2023-12-06​

  • Added input table used columns in lineage

[0.4.1] - 2023-11-09​

  • Alerts on "hanging" tasks
  • Adding users, pipeline owners and tags
  • Adding Spark resources metrics

[0.3.6] - 2023-10-11​

  • Time window filtering
  • Added Runs page
  • Alerts - show in groups, option to dismiss
  • Adding brush-and-zoom to chart
  • logic for "unknown" task statuses
  • Adding task level error
Last updated on