Skip to main content
Unlisted page
This page is unlisted. Search engines will not index it, and only users having a direct link can access it.

Server Changelog

Main functional features:

[0.60.4] - 2025-12-17​

New Features

  • Added dataset read/write API for programmatic data access
  • Added time-window filter to app runs API for more precise querying
  • Added health checks (readiness and liveness) to Helm chart
  • Added configurable serviceType for backend services in Helm chart (default: NodePort for ALB instance targeting)

Improvements

  • Redesigned task/PIT selectors with improved navigation and usability
  • Enhanced breadcrumbs with better task navigation and "All tasks" option
  • Improved comparative analysis form with synchronized environment and pipeline states
  • Fixed time filtering to use exact timestamps instead of day boundaries
  • Added partial retention for tasks to optimize storage
  • Migrated hotkey library from @react-hook/hotkey to react-hotkeys-hook
  • Updated frontend dependencies (React 19, MUI 7, Vite 7, Storybook 10)
  • Simplified query API with dedicated caller parameter
  • Removed legacy manual_type and manual_app_runs_status features

[0.60.3] - 2025-12-03​

New Features

  • Exposed Workload dashboard tab for cluster monitoring
  • Redesigned Datasets page with new design and improvements
  • Enhanced compare functionality with asset-level matching, fuzzy matching, and deduplication
  • Added version information in task-run details toolbox
  • in debug: Captured cost savings tracking and reporting
  • Improved Spark plan visualization with additional enhancements

Improvements

  • Changed default time window from 1 month to 1 week for faster initial load
  • Metric units now included in congestion API for clearer data interpretation
  • Added file grouping in Datasets table for better organization
  • Improved congestion component with dynamic metric data updates and performance optimizations
  • Better time window URL format for easier sharing and bookmarking
  • Enhanced LineChart performance for smoother visualizations
  • Fixed comparative-analysis params by swapping base and compared runs for correct comparison
  • Improved dataset info endpoint for better performance
  • Better error handling across the application
  • Default line color for Spark metrics charts
  • Optimized task run parameters in enrichments
  • Added input validation to global system settings
  • Reorganized sidebar navigation with improved item ordering and global background color
  • Fixed value histogram metric asset count in asset tree

[0.60.2] - 2025-11-27​

New Features

  • Added partition columns to dataset views for better data organization
  • Expanded dataset format support for improved compatibility

Improvements

  • Enhanced congestion metrics with additional monitoring options
  • Faster email notifications for pipeline completions

[0.60.1] - 2025-11-26​

New Features

  • Added dataset overview tab for quick dataset insights

Improvements

  • Improved congestion dashboard with sticky chart while scrolling
  • Enhanced backend task processing with async execution

[0.60.0] - 2025-11-23​

New Features

  • Introduced new Datasets table for comprehensive data management
  • Added Spark plan/code visualization to understand query execution

Improvements

  • Enhanced search with character highlighting for faster navigation
  • Improved tag filtering with better dropdown interface
  • Better system monitoring with new agent memory metrics

[0.59.1] - 2025-11-13​

Improvements

  • Performance optimizations across the platform

[0.59.0] - 2025-11-12​

New Features

  • Compute Congestion dashboard to identify and resolve resource bottlenecks
  • Single-run recommendations for one-time pipeline optimizations
  • Searchable compute entities for faster navigation

Improvements

  • More accurate insights impact forecasting using exponential decay
  • Insights automatically expand in pipelines and compute pages
  • Clickable tags with icons for easier filtering
  • Better data comparison with improved fuzzy matching
  • Faster compute query performance

[0.58.0] - 2025-11-03​

New Features

  • Global search bar for finding resources across the entire application
  • Spark logical plan parser to analyze query execution

Improvements

  • Redesigned compute view from task-level to pipeline-level perspective
  • Enhanced params page with automatic base-run selection
  • Improved code comparison with visual tree changes
  • Faster dashboard loading with optimized resource queries
  • Insight IDs now visible in dashboards for easier tracking
  • MCP prompts now include direct links in addition to copy functionality

[0.57.6] - 2025-10-24​

Improvements

  • Enhanced over-provisioned executors insight with filtering options
  • Improved static Databricks cluster insights
  • More accurate cost reporting in dashboards
  • Support for disabling specific tenants

[0.57.0] - 2025-10-15​

New Features

  • Dedicated compute dashboard with utilization metrics and cost tracking
  • Custom metrics support in asset tree and automated test generation
  • Dedicated Insights toolbox for centralized insight management
  • Cluster-to-task linking for better resource tracking

Improvements

  • Significantly faster chart rendering with canvas-based implementation
  • Improved browser navigation and history management
  • Extended default time window to 1 month for better historical context
  • MCP now includes lineage information
  • Enhanced orphaned vCores detection and recommendations

[0.56.3] - 2025-09-24​

  • Add agent rate-limit middleware
  • Enhance Retention policy
  • MCP support for streamable-http

[0.55.5] - 2025-09-09​

  • Initial MCP support

[0.54.2] - 2025-08-14​

  • Add license and usage
  • Add shuffle-partitions insight
  • Add s3 insight

[0.52.3] - 2025-07-20​

  • Improved support for Databricks clusters

[0.50.1] - 2025-07-16​

  • Huge UI improvements
    • Toolbox
  • Black-listed params
  • Error drill-down
  • Single task-run insights

[0.43.7] - 2025-05-25​

  • Added off-heap memory over-provisioning insight
  • On-prem SSO fix

[0.43.4] - 2025-05-09​

  • Added UI optimizations for tasks with many events
  • Improve date picker
  • Auth
    • Added microsoft auth
    • Added MFA login flow
    • Added refresh tokens support

[0.42.2] - 2025-04-08​

  • Dashboard zoom changes the time frame
  • Add active tasks metric
  • Add generic SSO support

[0.41.5] - 2025-03-25​

  • Fix for opt-in new pipelines
  • some UI fixes

[0.41.2] - 2025-03-14​

  • Auto tests generation improvements
  • A few bug fixes

[0.40.0] - 2025-03-09​

  • Added new on-boarding and sign-up flows
  • Added insights
    • many small files
    • skewed tasks
    • executors memory under-provisioning

[0.30.0] - 2025-02-23​

  • Added insights for idle/spill/GC time and driver unused heap
  • Health & cost dashboards improvements
  • Lineage improvements - unify data and execution, add task drill down

[0.20.3] - 2025-02-17​

  • hotfix for datasets with no schema

[0.20.2] - 2025-02-13​

  • Multi tenant issues hotfix

[0.20.1] - 2025-02-11​

  • Pipeline run notifications hotfix

[0.20.0] - 2025-01-29​

  • Major refactor of agent-server protocol (server is backward compatible)
  • Add support for pipeline-users permissions
  • Overall UI enhancements - layout, menus, etc.
  • Added time frame picker
  • Added run status of "Completed with retries"
  • Improved health dashboard
  • Improved timeline chart
  • Lineage improvements - support hopping to upstream pipelines
  • Fix wrong incidents for updating metrics

[0.15.0] - 2024-12-25​

  • Added pipeline and task insights and recommendations
  • Added events timeline to spark overview page
  • Improved health dashboard
  • Added simple retention policy
  • Added support for multi tenancy
  • Added SSO login
  • Added system setting to disable all agents

[0.14.7] - 2024-12-18​

  • Added parameters to ingress Helm chart

[0.14.6] - 2024-11-21​

  • Added SLA monthly digest email

[0.14.5] - 2024-11-21​

  • Added lineage view in compare-runs
  • Added code tab in compare-runs
  • Overall UI enhancements
  • Added cross task view in Spark-overview page
  • Fixed subscribed metrics report

[0.14.4] - 2024-11-06​

  • Enhancements to daily incidents mail
  • Add Code tab in compare-runs page
  • Add multi-thresholds in compare-runs page

[0.14.3] - 2024-11-02​

  • Add Freshness incidents to daily mail

[0.14.1] - 2024-10-27​

  • Compare-runs page enhancements
  • Fix daily SLA mail issue

[0.14.0] - 2024-10-10​

  • Added Comparative analysis page
  • Added SLA daily digest email

[0.13.0] - 2024-09-24​

  • Added incident entity to consolidate metric alerts
  • Added task idle time metric
  • Improved pipelines page

[0.12.0] - 2024-09-11​

  • Added pipelines coverage dashboard
  • Add more resource metrics to task and overtime - input/output/shuffle bytes, failures, retries, skew, broadcasts, gc, spill
  • Add support for run state management for agents

[0.11.2] - 2024-09-08​

  • Hotfix for a few edge cases

[0.11.0] - 2024-08-26​

  • Added Prophet based anomaly detection tests generator and dedicated pod/worker
  • Added Spark Driver memory timeseries metric
  • Added bulk actions option to metrics table
  • Added partition details the schema comparison page
  • Fix Spark metrics live updates
  • Enhanced the tags filter menu

[0.10.0] - 2024-07-31​

  • Added Pipeline resources and cost dashboards
  • Added Spark task resources overview page

[0.9.3] - 2024-07-08​

  • Fix handling sudden log spike

[0.9.2] - 2024-07-02​

  • Added new memory metrics - allocated, on-heap, off-heap, used, utilized for driver and executors
  • Email notification redesign
  • Added "reported metrics" section in run completed email notification

[0.9.1] - 2024-06-23​

  • Add support for files as pipeline inputs (with metrics)
  • Add "0"s in Histogram metric for values that were not reported
  • Overall many UI improvements
  • Improve metric values format depending on units
  • Improve alerts and dataset metrics pages
  • Optimize backend routes to handle higher load
  • Separate queries to another table to ease on DB export
  • Fix task duration bug

[0.8.4] - 2024-06-06​

  • Fix live data freshness metric issue
  • Fix adding missing values to histogram metric

[0.8.1] - 2024-06-05​

  • Hotfix slow heartbeat endpoint

[0.8.0] - 2024-06-02​

  • Added support for killing long running tasks ("break" on duration test)
  • Added support for notification per test (useful for Live Table Freshness metric)
  • Added user defined "Pipeline run ID" and "Task ID" identifiers
  • Unified metrics definitions across the system
  • Increased Histogram metric to 100 values
  • Improve auto tests

[0.7.6] - 2024-05-20​

  • Stability fixes
  • Enhance auto tests
  • Add live freshness metric for output tables
  • Fix alerts on "unknown" tasks
  • Improve UI responsiveness

[0.7.2] - 2024-04-19​

  • Add support for pipeline run id

[0.7.0] - 2024-04-15​

  • Add support for task-level metrics config
  • Add asset profiles to metric config
  • Add option to disable agent from UI
  • Improve email notification details
  • Improve metric names and detailed info
  • Improve auto tags logic

[0.6.4] - 2024-03-31​

  • Added support for metrics group view and tests
  • Added notification option for pipeline failed (including task retries)
  • Added metric count NaN values for numeric columns
  • Added lineage navigation in asset tree
  • Fixed missing metrics for input tables
  • Fixes for auto lineage logic
  • Fixes for auto generated tests
  • Api query optimizations

[0.6.3] - 2024-03-13​

  • Added suggested tests feature to replace "outdated" tests
  • Added notification for pipeline completed with alerts
  • Expand metrics asset tree component to all pages
  • Enable PIT navigation on the metric chart

[0.6.2] - 2024-03-07​

  • Added tests and alerts to metrics asset tree
  • Added notifications for task failed, task is hanging, pipeline completed, pipeline did not start
  • Show Spark's TF id in task page
  • Added system settings
  • Added pipeline settings
  • Added option to regenerate auto-tests per pipeline
  • Enhanced metric table - sorting, filtering, etc.
  • Added pipeline run metrics - count of tasks, cpu_time, used_cpu_time, process_time, duration, SLA
  • Fixed input table freshness metric

[0.5.2] - 2024-01-25​

  • Added Metrics - num TFs metric in task, num schema changes, task SLA
  • UI Improvements - Asset tree metrics component
  • Improved auto-tags logic
  • Added error tab in TF page

[0.5.0] - 2024-01-09​

  • Task params comparison
  • Dataset schema comparison
  • Added dedicated test drawer
  • Page titles
  • Dataset downstream tab
  • Added auto pipeline tags logics
  • Docs link in UI
  • Metric chart improvements
  • Added server version to task params
  • Cleaning Spark's query plan from temporary column numbers

[0.4.2] - 2023-12-06​

  • Added input table used columns in lineage

[0.4.1] - 2023-11-09​

  • Alerts on "hanging" tasks
  • Adding users, pipeline owners and tags
  • Adding Spark resources metrics

[0.3.6] - 2023-10-11​

  • Time window filtering
  • Added Runs page
  • Alerts - show in groups, option to dismiss
  • Adding brush-and-zoom to chart
  • logic for "unknown" task statuses
  • Adding task level error
Last updated on