Skip to main content
Unlisted page
This page is unlisted. Search engines will not index it, and only users having a direct link can access it.

Server Changelog

Main functional features:

[0.68.0] - 2026-03-04​

New Features​

  • Added Executor Off-Heap Memory time series chart for Spark tasks (#4454)
  • Added Dataproc Serverless detection as a distinct task sub-type for accurate classification (#4448)
  • Added audit logging framework for authentication events (#4433)
  • Added support for filtering pipelines by tag names in app filter (#4425)
  • Added auto-tag for disabled pipelines (#4444)

Improvements​

  • Optimized backend speed for incidents API with significant performance gains (#4378)
  • TF code plan design improvements for better readability (#4410)
  • Added rootTfId-based stage filtering for more granular Spark stage views (#4430)
  • Return descriptive error messages instead of generic 403 for auth failures (#4372, #4413)
  • Dark mode visual improvements across multiple components (#4402)
  • Added bigint limit validation to agent task ID fields (#4431)

Bug Fixes​

  • Fixed insights executor config display logic using is_dynamic flag (#4450)
  • Fixed app filter returning all results for invalid tag/owner values (#4443)
  • Fixed Compare Runs Params table: swapped values, form auto-fill, and visual width (#4440)
  • Fixed Datasets collapse-all button (#4424)
  • Fixed cost dashboard making double API requests for resources (#4422)

[0.67.0] - 2026-02-22​

New Features​

  • Enhanced AI prompts with structured workflows, new skills (/compare, /validate-release, /investigate-alert), and comprehensive metrics reference guide (#4351)
  • Added timeline range minimap controller for better visualization of zoomed chart areas with drag navigation (#4377)
  • Added AI Assistant entity mentions with @ autocomplete for pipelines, tasks, datasets, and computes (#4311)
  • Added tenant setting to require admin approval for OAuth/SSO users for enhanced security control (#4312)
  • Added skills system to AI Assistant with on-demand skill loading and tool call indicators (#4274)
  • Added regex pattern matching for time series metrics with dynamic color assignment and asset-based row IDs (#4202)
  • Pipeline settings redesigned with new reusable Modal, NavContent, and ModalWithNavigation components (#4354)
  • Environment selector now defaults to environment with most pipelines and includes search for 6+ environments (#4299)
  • Pipelines now marked as deleted before background deletion starts (#4142)

Improvements​

  • Dashboard GroupBy now works in conjunction with appTags and migrated to URL-search param based solution (#4393)
  • Improved loading UX by eliminating global page loader and using component-level loading states to prevent cache invalidation (#4383)
  • Made incidents API app_id optional to support cross-environment queries (35ms for 8K incidents) with backward compatibility (#4384)
  • Improved Spark metric color assignment using priority-based indexing for consistent color distribution across priority groups (#4380)
  • Add separate SMTP login email setting to allow different authentication and sender addresses (#4390)
  • Add configurable tenant-level insight thresholds (min_runs, min_days, min_cost) with centralized annual impact calculation (#4317)
  • Refactored time-series metrics color system to priority-based frontend logic for better maintainability (#4355)
  • Improved resizable panels with 8 bug fixes including StrictMode compatibility and window resize handling (#4373)
  • AI prompts enhancements: skill descriptions, template-driven system prompt, and compare-runs page context (#4328)
  • Spark page now supports resizing between chart/timeline and chart/table with improved UX (#4316)
  • Hide chart toggles and legend items when no data available for cleaner UI (#4382)
  • Added configurable insights generation interval system setting (default: 3 days) (#4310)
  • Added automatic tag extraction for spark.yarn.queue parameter (#4363)
  • Refactored query hooks to makeQuery API for cleaner, more maintainable code (#4285)
  • Improved auth flow with pending tenant notification fix and comprehensive test refactoring (#4338)
  • Improved AI context with time window formatting and incident start pit field (#4245)
  • AI Assistant base context chips now display page, env, pipeline, time window, and tab information (#4311)
  • Monaco editor shortcut whitelist to prevent conflicts with AI chat input (#4311)
  • Improved SMTP error handling with clear user-facing messages (#4342)
  • Switched task run status display to show "with retries" on top-level row (#4341)
  • Added partial index to optimize over-provisioned executors insight query performance (#4339)
  • Stress test improvements: auto-generate jsonlines, tenant support, and ID fixes (#4356)
  • Updated Biome to latest version with improved linting rules (#4350)
  • Refactored log API tests for better organization (#4336)
  • Cleaned up DefComps and refactored Runs page for better maintainability (#4347)
  • Removed sql- prefix from server timing headers for cleaner logging (#4304)
  • Minor packages update with various bugfixes (#4286)
  • Update tf_inputs columns on conflict instead of ignoring for better data consistency (#4292)

Bug Fixes​

  • Fixed Spark overview flexbox panels failing to initialize on slow networks by replacing polling with sync-on-every-render (#4395)
  • Fixed insight drawer navigation with multi-insight URL synchronization and stable pagination (#4329, #4368)
  • Fixed cost tab to show all tasks for pipelines and logical tasks (#4381)
  • Fixed auth endpoints to use write DB instead of read replicas, resolving login failures (#4327)
  • Fixed first OAuth/SSO user not becoming tenant admin for new tenants (#4335)
  • Fixed bottom axis tick count to be deterministic across renders, eliminating e2e test flakiness (#4332)
  • Fixed stale data rendering in Pipeline and Task overview components (#4334)
  • Fixed None unit display in alert emails and added asset_value display (#4313)
  • Fixed SSO MFA KeyError for tenant_name on new user creation (#4307)
  • Fixed cost settings cache bug by marking functions as STABLE (#4305)
  • Fixed alerts test by waiting for loading indicators (#4302)
  • Fixed PostHog configuration issues (#4276)
  • Fixed drop view error in read-only transaction for computes API (#4293)
  • Dropped duplicate foreign key constraint on users.tenant_id (#4343)

[0.66.2] - 2026-02-08​

Improvements​

  • Separate all 4xx error metrics with thread-safe counter creation for better monitoring (#4271)
  • Add configurable batch size limit for unresponsive tasks to improve performance (#4280)
  • AI chat improvements with enhanced functionality (#4262)
  • Added debounce to plan timeline tooltips for better UX (#4269)
  • Allow pipeline_run_id to take precedence over app_pit for more accurate tracking (#4260)
  • Add asset_type to time-series metrics for improved data classification (#4251)
  • Add Claude Code commands for quick-pr and fix-alembic workflows (#4267)
  • Add covering index test for task_enrichments to improve query performance (#4261)
  • Added e2e tests for compare pipelines, lineage in compare metrics tab, and compare-runs form (#4246, #4254, #4277)

Bug Fixes​

  • Fix chunk load errors after deployments with auto-reload (#4266)
  • Fix missing task-params in feature generation pipeline (#4265)
  • Read replica fixes: per-route write DB dependencies (#4253, #4250)
  • Fix compare lineage paths to use original asset names instead of shadow names (#4239)
  • Hide is_debug_mode parameter from Swagger documentation (#4272)
  • Compare-runs form bugfixes (#4277)

[0.66.1] - 2026-02-01​

Improvements​

  • Save filters (timeWindow, appTags & appOwners) to browser storage, per-env (#4242)
  • Task params now updated when switching between retry runs (#4244)
  • Added e2e test for task dropdown with/without run selection (#4241)
  • Added e2e test for compare runs with task selection (#4240)

Bug Fixes​

  • Fixed swapping compared-runs in comparative-analysis (#4237)
  • Prevent overwriting COMPLETED transformation status (#4235)
  • Fix prompt edit/delete failure for names with special characters (#4233)
  • Revert is_saas/admin features decoupling due to issues (#4238)

[0.66.0] - 2026-01-28​

New Features​

  • Added read replica support for database scaling and improved read performance (#4213)
  • Added OpenTelemetry distributed tracing (OTLP) for enhanced observability (#4208)
  • Added e2e tests for insights drawer covering all 15 insight types (#4217)
  • Added TF plan timeline item tooltips for better context (#4214)

Improvements​

  • AI Chat Assistant Enhancements: chat history, session management, improved styling
  • Enhanced PostHog configuration with better user tracking and analytics (#4119)
  • Improved system settings with clearer titles and descriptions (#4195)
  • Simplified license system by removing from AccessToken and adding graceful expiration handling (#3998)
  • Routine frontend packages update for security and performance (#4197)
  • Enhanced e2e test database coverage with multiple updates (#4207, #4162, #4154, #4198, #4167)
  • Updated demo with code improvements insights (#4206)
  • Decoupled is_saas from admin features in SuperAdmin UI for proper access control (#4226)
  • Disabled e2e task retention for improved test performance (#4228)
  • Optimized Postgres performance with cached register_types (#4223)

Bug Fixes​

  • Fixed /api/tags to return schedule tags for all tenants (#4210)
  • Fixed is_mfa default to false and prevent test from overwriting settings (#4209)
  • Deleted duplicate foreign key in metrics_conf table (#4203)
  • Fixed plan timeline tooltip to handle undefined metadata gracefully (#4225)
  • Fixed slash commands in AI assistant (#4223)

[0.65.0] - 2026-01-25​

New Features​

  • Added MFA support for Google/Microsoft SSO login (#4186)
  • Added system-wide SSO MFA setting for centralized authentication control (#4193)
  • Added MCP dataset tools with UI improvements for enhanced data management (#4164)
  • Added retry tracking to task aggregate runs view for better visibility (#4176)

Improvements​

  • Redesigned PIT selector options list with improved previous-runs items navigation (#4191)
  • Major TF plan refactor for better performance and maintainability (#4160)
  • Improved SparkInsight UI layout with overflow handling (#4192)
  • Color stage timeline by vCore time instead of duration for more accurate resource visualization (#4177)
  • Added app_id and task_sub_type to task-run search APIs for enhanced filtering (#4178)
  • Added sub_category filter in TF events API for physical plans (#4188)
  • Removed redundant compute cluster name references for cleaner codebase (#4180)
  • Replaced z-max with semantic z-index utilities for better CSS maintainability (#4189)
  • AI toolbox now closes on page load by default (#4187)
  • Added various SKILL files to Claude folder for AI assistant improvements (#4183)

Bug Fixes​

  • Fixed chart selection brush issue with additional refactors (#4190)
  • Fixed global search "TF" item visual issue (#4185)
  • Fixed typo in INCLUDE_SOURCEMAPS environment variable (#4168)

[0.64.2] - 2026-01-20​

Improvements​

  • Replaced pipeline completion notifications with task completion notifications for more granular control (#4158)
  • Improved count of environment objects for better performance (#4159)
  • Enhanced multi-tenant support in stress tests (#4152)
  • Refactored login page business logic into hooks and utilities for better code organization (#4147)
  • Deprecated react-resizable-panels in favor of custom usePanelResize hook (#4148)
  • Enhanced dark-theme visual consistency across multiple components (#4150)

Bug Fixes​

  • Fixed MCP authorization for Pydantic model arguments (#4163)
  • Fixed styling bug with compute table stars (favorites) (#4156)

[0.64.0] - 2026-01-18​

New Features​

  • Dark mode support (#3988)
  • Added WorkOS SSO integration for enterprise single sign-on authentication (#4141)
  • Added lightweight /tasks-basic endpoint for optimized task selectors and dropdowns (#4141)
  • Consolidated SuperAdmin pages into tabs and removed from sidebar (#4139)
  • Added Settings tab to SuperAdmin page (#4117)
  • Enabled AI Assistant based on Anthropic Key and Tenant Settings (#4079)

Improvements​

  • Refactored useTrackRecentObject hook for better performance and stability (#4145)
  • Improved two-step login flow with email-first approach for better UX (#4141)
  • Enhanced dark-mode visual consistency across components (#4145)
  • Optimized task queries using JSONB and improved null handling (#4141)
  • Optimized slow SQL queries for better performance (#4138)
  • Consolidated page-level tracking logic into dedicated hooks for better code organization (#4145)
  • Added GlobalSearchAssetChip component for improved search modal display (#4145)
  • Extracted route matchers for better code reuse (#4145)
  • Refactored toolbox-related code and allow resizing fluidly (#4122)
  • Added intelligent caching to BaseResourceResolver for security performance (#4123)
  • Refactored Spark plan handling: increased limit and simplified logic (#4124)
  • Refactored pipeline_dal to consolidate app_id lookup functions (#4110)
  • Refactored /api/compare routes to use functional parameters (#4098)
  • Refactored env endpoints and fixed null handling for environments without tasks (#4118)
  • Enabled self-monitoring export and fixed stress test dataset endpoints (#4109)
  • Removed redundant app_id and app_name params from task-runs/basic endpoint (#4140)
  • Allow special characters in URL (#4137)

Bug Fixes​

  • Fixed small bug in task-retries insight (#4120)

[0.63.2] - 2026-01-12​

New Features​

  • Added recent objects feature to search modal (#4086)

Improvements​

  • RESTful API routes refactoring for datasets, time-series-metrics, lineage, search (#4089)
  • Enhanced dataset format column to show actual file formats (#4088)
  • Improved TF table with input cell links and other fixes (#4083)
  • Use SecretStr for sensitive settings to prevent secret leakage in logs (#4082)

Bug Fixes​

  • Fixed dataset API to use query parameters for ds_name (#4105)
  • Fixed cnt_values metrics in compare when both sides are None (#4097)
  • Fixed alert sending from "other" metrics (#4096)
  • Fixed chart content updates when changing environment (#4092)

[0.63.1] - 2026-01-08​

New Features​

  • Added /tasks/{task_name}/runs API endpoint for task run queries (#4080)

Improvements​

  • Added created/updated timestamps to edit test drawer (#4074)

Bug Fixes​

  • Fixed metric_value always null in task metrics endpoint (#4077)

[0.62.1] - 2025-12-30​

New Features​

  • Added license management UI to super-admin page
  • Added MCP tools for transformation (TF) operations
  • Improved user management with status field and approval workflow
  • Added environment info to security resolvers and AI tools

Improvements​

  • Upgraded react-router from v6 to v7
  • Upgraded Tailwind from v3 to v4
  • Refactored metrics to use DELTA aggregation with aggregated SQL metrics
  • Tasks column items in Metrics table are now navigable
  • Agent requests are now rejected early when pipeline is disabled
  • Added support for imagePullSecrets in Helm chart
  • Consolidated small tenant-scoped routers for cleaner code organization

Bug Fixes​

  • Fixed security vulnerabilities in Python and JS dependencies
  • Fixed response body logging for /log routes
  • Fixed TF duration to preserve milliseconds precision
  • Fixed SLA metric bug
  • Fixed time-series metrics bucket size updates
  • Fixed migration failures with CASCADE on DROP VIEW statements

[0.61.0] - 2025-12-24​

New Features​

  • Added GenAI internal assistant (v1)
  • Added backend service metrics with OpenTelemetry
  • Added support for gzip content requests
  • New enhanced dataset page

Improvements​

  • Server state now takes precedence for agent enabled flag
  • Fixed metrics retention query performance and added FK constraint
  • Fixed warning color in notification emails to match app theme
  • Fixed timeline item highlighting when clicking TF
  • Removed false "related compute" items in toolbox
  • Removed redundant app_id column from events table
  • Improved PostHog analytics configuration with batching, privacy compliance, and cost optimization

[0.60.4] - 2025-12-17​

New Features​

  • Added dataset read/write API for programmatic data access
  • Added time-window filter to app runs API for more precise querying
  • Added health checks (readiness and liveness) to Helm chart
  • Added configurable serviceType for backend services in Helm chart (default: NodePort for ALB instance targeting)

Improvements​

  • Redesigned task/PIT selectors with improved navigation and usability
  • Enhanced breadcrumbs with better task navigation and "All tasks" option
  • Improved comparative analysis form with synchronized environment and pipeline states
  • Fixed time filtering to use exact timestamps instead of day boundaries
  • Added partial retention for tasks to optimize storage
  • Migrated hotkey library from @react-hook/hotkey to react-hotkeys-hook
  • Updated frontend dependencies (React 19, MUI 7, Vite 7, Storybook 10)
  • Simplified query API with dedicated caller parameter
  • Removed legacy manual_type and manual_app_runs_status features

[0.60.3] - 2025-12-03​

New Features​

  • Exposed Workload dashboard tab for cluster monitoring
  • Redesigned Datasets page with new design and improvements
  • Enhanced compare functionality with asset-level matching, fuzzy matching, and deduplication
  • Added version information in task-run details toolbox
  • in debug: Captured cost savings tracking and reporting
  • Improved Spark plan visualization with additional enhancements

Improvements​

  • Changed default time window from 1 month to 1 week for faster initial load
  • Metric units now included in congestion API for clearer data interpretation
  • Added file grouping in Datasets table for better organization
  • Improved congestion component with dynamic metric data updates and performance optimizations
  • Better time window URL format for easier sharing and bookmarking
  • Enhanced LineChart performance for smoother visualizations
  • Fixed comparative-analysis params by swapping base and compared runs for correct comparison
  • Improved dataset info endpoint for better performance
  • Better error handling across the application
  • Default line color for Spark metrics charts
  • Optimized task run parameters in enrichments
  • Added input validation to global system settings
  • Reorganized sidebar navigation with improved item ordering and global background color
  • Fixed value histogram metric asset count in asset tree

[0.60.2] - 2025-11-27​

New Features​

  • Added partition columns to dataset views for better data organization
  • Expanded dataset format support for improved compatibility

Improvements​

  • Enhanced congestion metrics with additional monitoring options
  • Faster email notifications for pipeline completions

[0.60.1] - 2025-11-26​

New Features​

  • Added dataset overview tab for quick dataset insights

Improvements​

  • Improved congestion dashboard with sticky chart while scrolling
  • Enhanced backend task processing with async execution

[0.60.0] - 2025-11-23​

New Features​

  • Introduced new Datasets table for comprehensive data management
  • Added Spark plan/code visualization to understand query execution

Improvements​

  • Enhanced search with character highlighting for faster navigation
  • Improved tag filtering with better dropdown interface
  • Better system monitoring with new agent memory metrics

[0.59.1] - 2025-11-13​

Improvements​

  • Performance optimizations across the platform

[0.59.0] - 2025-11-12​

New Features​

  • Compute Congestion dashboard to identify and resolve resource bottlenecks
  • Single-run recommendations for one-time pipeline optimizations
  • Searchable compute entities for faster navigation

Improvements​

  • More accurate insights impact forecasting using exponential decay
  • Insights automatically expand in pipelines and compute pages
  • Clickable tags with icons for easier filtering
  • Better data comparison with improved fuzzy matching
  • Faster compute query performance

[0.58.0] - 2025-11-03​

New Features​

  • Global search bar for finding resources across the entire application
  • Spark logical plan parser to analyze query execution

Improvements​

  • Redesigned compute view from task-level to pipeline-level perspective
  • Enhanced params page with automatic base-run selection
  • Improved code comparison with visual tree changes
  • Faster dashboard loading with optimized resource queries
  • Insight IDs now visible in dashboards for easier tracking
  • MCP prompts now include direct links in addition to copy functionality

[0.57.6] - 2025-10-24​

Improvements​

  • Enhanced over-provisioned executors insight with filtering options
  • Improved static Databricks cluster insights
  • More accurate cost reporting in dashboards
  • Support for disabling specific tenants

[0.57.0] - 2025-10-15​

New Features​

  • Dedicated compute dashboard with utilization metrics and cost tracking
  • Custom metrics support in asset tree and automated test generation
  • Dedicated Insights toolbox for centralized insight management
  • Cluster-to-task linking for better resource tracking

Improvements​

  • Significantly faster chart rendering with canvas-based implementation
  • Improved browser navigation and history management
  • Extended default time window to 1 month for better historical context
  • MCP now includes lineage information
  • Enhanced orphaned vCores detection and recommendations

[0.56.3] - 2025-09-24​

  • Add agent rate-limit middleware
  • Enhance Retention policy
  • MCP support for streamable-http

[0.55.5] - 2025-09-09​

  • Initial MCP support

[0.54.2] - 2025-08-14​

  • Add license and usage
  • Add shuffle-partitions insight
  • Add s3 insight

[0.52.3] - 2025-07-20​

  • Improved support for Databricks clusters

[0.50.1] - 2025-07-16​

  • Huge UI improvements
    • Toolbox
  • Black-listed params
  • Error drill-down
  • Single task-run insights

[0.43.7] - 2025-05-25​

  • Added off-heap memory over-provisioning insight
  • On-prem SSO fix

[0.43.4] - 2025-05-09​

  • Added UI optimizations for tasks with many events
  • Improve date picker
  • Auth
    • Added microsoft auth
    • Added MFA login flow
    • Added refresh tokens support

[0.42.2] - 2025-04-08​

  • Dashboard zoom changes the time frame
  • Add active tasks metric
  • Add generic SSO support

[0.41.5] - 2025-03-25​

  • Fix for opt-in new pipelines
  • some UI fixes

[0.41.2] - 2025-03-14​

  • Auto tests generation improvements
  • A few bug fixes

[0.40.0] - 2025-03-09​

  • Added new on-boarding and sign-up flows
  • Added insights
    • many small files
    • skewed tasks
    • executors memory under-provisioning

[0.30.0] - 2025-02-23​

  • Added insights for idle/spill/GC time and driver unused heap
  • Health & cost dashboards improvements
  • Lineage improvements - unify data and execution, add task drill down

[0.20.3] - 2025-02-17​

  • hotfix for datasets with no schema

[0.20.2] - 2025-02-13​

  • Multi tenant issues hotfix

[0.20.1] - 2025-02-11​

  • Pipeline run notifications hotfix

[0.20.0] - 2025-01-29​

  • Major refactor of agent-server protocol (server is backward compatible)
  • Add support for pipeline-users permissions
  • Overall UI enhancements - layout, menus, etc.
  • Added time frame picker
  • Added run status of "Completed with retries"
  • Improved health dashboard
  • Improved timeline chart
  • Lineage improvements - support hopping to upstream pipelines
  • Fix wrong incidents for updating metrics

[0.15.0] - 2024-12-25​

  • Added pipeline and task insights and recommendations
  • Added events timeline to spark overview page
  • Improved health dashboard
  • Added simple retention policy
  • Added support for multi tenancy
  • Added SSO login
  • Added system setting to disable all agents

[0.14.7] - 2024-12-18​

  • Added parameters to ingress Helm chart

[0.14.6] - 2024-11-21​

  • Added SLA monthly digest email

[0.14.5] - 2024-11-21​

  • Added lineage view in compare-runs
  • Added code tab in compare-runs
  • Overall UI enhancements
  • Added cross task view in Spark-overview page
  • Fixed subscribed metrics report

[0.14.4] - 2024-11-06​

  • Enhancements to daily incidents mail
  • Add Code tab in compare-runs page
  • Add multi-thresholds in compare-runs page

[0.14.3] - 2024-11-02​

  • Add Freshness incidents to daily mail

[0.14.1] - 2024-10-27​

  • Compare-runs page enhancements
  • Fix daily SLA mail issue

[0.14.0] - 2024-10-10​

  • Added Comparative analysis page
  • Added SLA daily digest email

[0.13.0] - 2024-09-24​

  • Added incident entity to consolidate metric alerts
  • Added task idle time metric
  • Improved pipelines page

[0.12.0] - 2024-09-11​

  • Added pipelines coverage dashboard
  • Add more resource metrics to task and overtime - input/output/shuffle bytes, failures, retries, skew, broadcasts, gc, spill
  • Add support for run state management for agents

[0.11.2] - 2024-09-08​

  • Hotfix for a few edge cases

[0.11.0] - 2024-08-26​

  • Added Prophet based anomaly detection tests generator and dedicated pod/worker
  • Added Spark Driver memory timeseries metric
  • Added bulk actions option to metrics table
  • Added partition details the schema comparison page
  • Fix Spark metrics live updates
  • Enhanced the tags filter menu

[0.10.0] - 2024-07-31​

  • Added Pipeline resources and cost dashboards
  • Added Spark task resources overview page

[0.9.3] - 2024-07-08​

  • Fix handling sudden log spike

[0.9.2] - 2024-07-02​

  • Added new memory metrics - allocated, on-heap, off-heap, used, utilized for driver and executors
  • Email notification redesign
  • Added "reported metrics" section in run completed email notification

[0.9.1] - 2024-06-23​

  • Add support for files as pipeline inputs (with metrics)
  • Add "0"s in Histogram metric for values that were not reported
  • Overall many UI improvements
  • Improve metric values format depending on units
  • Improve alerts and dataset metrics pages
  • Optimize backend routes to handle higher load
  • Separate queries to another table to ease on DB export
  • Fix task duration bug

[0.8.4] - 2024-06-06​

  • Fix live data freshness metric issue
  • Fix adding missing values to histogram metric

[0.8.1] - 2024-06-05​

  • Hotfix slow heartbeat endpoint

[0.8.0] - 2024-06-02​

  • Added support for killing long running tasks ("break" on duration test)
  • Added support for notification per test (useful for Live Table Freshness metric)
  • Added user defined "Pipeline run ID" and "Task ID" identifiers
  • Unified metrics definitions across the system
  • Increased Histogram metric to 100 values
  • Improve auto tests

[0.7.6] - 2024-05-20​

  • Stability fixes
  • Enhance auto tests
  • Add live freshness metric for output tables
  • Fix alerts on "unknown" tasks
  • Improve UI responsiveness

[0.7.2] - 2024-04-19​

  • Add support for pipeline run id

[0.7.0] - 2024-04-15​

  • Add support for task-level metrics config
  • Add asset profiles to metric config
  • Add option to disable agent from UI
  • Improve email notification details
  • Improve metric names and detailed info
  • Improve auto tags logic

[0.6.4] - 2024-03-31​

  • Added support for metrics group view and tests
  • Added notification option for pipeline failed (including task retries)
  • Added metric count NaN values for numeric columns
  • Added lineage navigation in asset tree
  • Fixed missing metrics for input tables
  • Fixes for auto lineage logic
  • Fixes for auto generated tests
  • Api query optimizations

[0.6.3] - 2024-03-13​

  • Added suggested tests feature to replace "outdated" tests
  • Added notification for pipeline completed with alerts
  • Expand metrics asset tree component to all pages
  • Enable PIT navigation on the metric chart

[0.6.2] - 2024-03-07​

  • Added tests and alerts to metrics asset tree
  • Added notifications for task failed, task is hanging, pipeline completed, pipeline did not start
  • Show Spark's TF id in task page
  • Added system settings
  • Added pipeline settings
  • Added option to regenerate auto-tests per pipeline
  • Enhanced metric table - sorting, filtering, etc.
  • Added pipeline run metrics - count of tasks, cpu_time, used_cpu_time, process_time, duration, SLA
  • Fixed input table freshness metric

[0.5.2] - 2024-01-25​

  • Added Metrics - num TFs metric in task, num schema changes, task SLA
  • UI Improvements - Asset tree metrics component
  • Improved auto-tags logic
  • Added error tab in TF page

[0.5.0] - 2024-01-09​

  • Task params comparison
  • Dataset schema comparison
  • Added dedicated test drawer
  • Page titles
  • Dataset downstream tab
  • Added auto pipeline tags logics
  • Docs link in UI
  • Metric chart improvements
  • Added server version to task params
  • Cleaning Spark's query plan from temporary column numbers

[0.4.2] - 2023-12-06​

  • Added input table used columns in lineage

[0.4.1] - 2023-11-09​

  • Alerts on "hanging" tasks
  • Adding users, pipeline owners and tags
  • Adding Spark resources metrics

[0.3.6] - 2023-10-11​

  • Time window filtering
  • Added Runs page
  • Alerts - show in groups, option to dismiss
  • Adding brush-and-zoom to chart
  • logic for "unknown" task statuses
  • Adding task level error
Last updated on