Feature
New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions:
New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions:
- 2.1.119-debian11, 2.1.119-rocky8, 2.1.119-ubuntu20, 2.1.119-ubuntu20-arm
- 2.2.87-debian12, 2.2.87-rocky9, 2.2.87-ubuntu22, 2.2.87-ubuntu22-arm
- 2.3.36-debian12, 2.3.36-ml-ubuntu22, 2.3.36-rocky9, 2.3.36-ubuntu22, 2.3.36-ubuntu22-arm
- 3.0.2-debian13, 3.0.2-ml-ubuntu24, 3.0.2-rocky9, 3.0.2-ubuntu24
Key updates in these image versions include:
- Feature updates:
- Apache Hudi: Added support for the Apache Hudi optional component in 3.0 images. Version 1.2.0 is available in 3.0 images.
- Parquet footer caching: Enabled Parquet footer caching by default for Lightning Engine (Velox). If executor out-of-memory (OOM) or task failure spikes on tiny files are observed, these can be mitigated by setting spark.gluten.sql.columnar.backend.velox.cacheParquetFooters=false.
- Apache Iceberg 1.10: Added support for Apache Iceberg 1.10 in 2.2 images. Users can opt-in by setting the cluster property, dataproc:dataproc.iceberg.version=1.10.
- Lakehouse catalog: Lakehouse catalog auto-loading is supported for image versions 2.2 and later.
- Library updates:
- Cloud Storage connector: Upgraded the Cloud Storage connector to 4.0.4 in 3.0 images.
- OpenLineage: Upgraded OpenLineage to 1.49 in 3.0 images to support lineage for tables created using the Lakehouse Runtime catalog.
Read the original announcement