Feature
New Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark) subminor runtime versions:
New Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark) subminor runtime versions:
- 1.2.89
- 2.2.89
- 2.3.42
Key updates in these runtime versions include:
- Feature updates:
- Apache Iceberg 1.10: In the 2.3 runtime, Serverless 2.3 batches with Lightning Engine now use Apache Iceberg version 1.10 by default.
- Parquet footer caching: Enabled Parquet footer caching by default for Lightning Engine (Velox) in the 2.3 runtime. If executor out-of-memory (OOM) or task failure spikes on tiny files are observed, these can be mitigated by setting spark.gluten.sql.columnar.backend.velox.cacheParquetFooters=false.
- Lakehouse catalog: Enabled Lakehouse catalog auto-loading in the 2.3 runtime using the spark.dataproc.lakehouse.project property.
Read the original announcement