Managed Spark and Hadoop

Public Preview: Azure Databricks Private network gateway

Private network gateway for Azure Databricks is now in Public Preview, enabling serverless compute to connect to resources in your virtual network (VNet) and transitively connected networks through a single managed gateway.

Amazon EMR now supports native data quality checks in Apache Spark jobs

Amazon EMR now validates data quality natively within Apache Spark jobs. Data engineering teams can catch quality issues inline before bad data reaches downstream systems, without building or maintaining separate validation infrastructure.

New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions:

2.1.120-debian11, 2.1.120-rocky8, 2.1.120-ubuntu20, 2.1.120-ubuntu20-arm

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): A bug has been fixed to add-back py4j library in image version 2.1.120.

Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark): Users may now specify short template names when creating a serverless…

Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark): Users may now specify short template names when creating a serverless session.

Serverless Storage on Amazon EMR Serverless now supports terabyte-scale shuffle

Amazon EMR Serverless now offers enhanced serverless storage capabilities with support for up to 1TB shuffle operations, raising the previous 200 GB per-job limit.

Apache Iceberg materialized views now support system-managed write protection

Today, AWS announces system-managed materialized views for Apache Iceberg. Materialized views let you precompute an expensive query once and reuse the result across engines.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine):

End-to-End Encryption for Apache Ranger (HTTPS): Ranger Admin, UserSync, and client plugins now enforce HTTPS on port 6182 for SSL-enabled clusters.

New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions:

2.2.88-debian12, 2.2.88-rocky9, 2.2.88-ubuntu22, 2.2.88-ubuntu22-arm

Run interactive workloads on Amazon EMR on EKS with Spark Connect

Amazon EMR on EKS now supports interactive Apache Spark sessions with Spark Connect. Data engineers and data scientists can develop and debug Apache Spark applications interactively from managed notebooks in Amazon SageMaker Unified Studio and their own IDEs, such as Jupyter and…

Amazon EMR on EKS now supports IPv6 Amazon EKS clusters

Today, AWS announces that Amazon EMR on EKS now supports running workloads on IPv6 Amazon Elastic Kubernetes Service (Amazon EKS) clusters.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Lightning Engine is now supported in 3.0 cluster image versions starting with subminor…

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Lightning Engine is now supported in 3.0 cluster image versions starting with subminor image version 3.0.2 ( 3.0.2-debian13 and 3.0.2-ubuntu24 ).

Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark):

Preconfigured Conda channels removed: Preconfigured conda channels (such as conda-forge ) have been removed from configurations. All runtime version aliases now point to the latest runtime version without conda channel configuration.

New Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark) subminor runtime versions:

1.2.89

Amazon EMR 7.14 is now available

Amazon EMR 7.14 is now available with new features across Amazon EMR on EC2, Amazon EMR on EKS, and Amazon EMR Serverless, along with version upgrades for additional applications. This release also upgrades Apache Spark to 3.5.8 and Apache Iceberg to 1.10.1.

Amazon EMR introduces Long Term Support with Apache Spark 4.1

Amazon EMR introduces Long Term Support (LTS) releases, starting with emr-spark-8.1.0 and Apache Spark 4.1. With LTS, designated versions of the AWS runtime for Apache Spark receive 36 months of support.

Amazon EMR extends support at no additional cost for customers migrating off older releases

Amazon EMR now gives you more time, at no additional cost, to migrate off older releases before they reach end of support.

Managed Service for Apache Spark (formerly Dataproc on Compute Engine):

Preconfigured Conda channels removed: Preconfigured conda channels (such as conda-forge ) have been removed from configurations. All image version aliases now point to the latest image without conda channel configuration.

New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions:

2.1.119-debian11, 2.1.119-rocky8, 2.1.119-ubuntu20, 2.1.119-ubuntu20-arm

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Fixed a segmentation fault when OpenLineage parses complex SQL query strings with the…

Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Fixed a segmentation fault when OpenLineage parses complex SQL query strings with the Lakehouse Runtime catalog.