Release 468 (17 Dec 2024)#

General#

  • Add new node states DRAINING and DRAINED to make it possible to reactivate a draining worker node. (#24444)

BigQuery connector#

  • Improve performance when reading external BigLake tables. (#21016)

Delta Lake connector#

  • ⚠️ Reduce coordinator memory usage for the Delta table metadata cache and enable configuration delta.metadata.cache-max-retained-size to control memory usage. Remove the configuration property delta.metadata.cache-size and increase the default for delta.metadata.cache-ttl to 30m. (#24432)

Hive connector#

  • Enable mismatched bucket execution optimization by default. This can be disabled with hive.optimize-mismatched-bucket-count configuration property and the optimize_mismatched_bucket_count session property. (#23432)

  • Improve performance by deactivating bucket execution when not useful in query processing. (#23432)

Iceberg connector#

  • Improve performance when running a join or aggregation on a bucketed table with bucketed execution. This can be deactivated with the iceberg.bucket-execution configuration property and the bucket_execution_enabled session property. (#23432)

  • Deprecate the iceberg.materialized-views.storage-schema configuration property. (#24398)

  • ⚠️ Rename the partitions column in the $manifests metadata table to partition_summaries. (#24103)

  • Avoid excessive resource usage on coordinator when reading Iceberg system tables. (#24396)

PostgreSQL connector#

SPI#

  • Add partitioning push down, which a connector can use to activate optional partitioning or choose between multiple partitioning strategies. (#23432)