Amazon EMR

Amazon EMR now supports native data quality checks in Apache Spark jobs

Amazon EMR now validates data quality natively within Apache Spark jobs. Data engineering teams can catch quality issues inline before bad data reaches downstream systems, without building or maintaining separate validation infrastructure.

Serverless Storage on Amazon EMR Serverless now supports terabyte-scale shuffle

Amazon EMR Serverless now offers enhanced serverless storage capabilities with support for up to 1TB shuffle operations, raising the previous 200 GB per-job limit.

Apache Iceberg materialized views now support system-managed write protection

Today, AWS announces system-managed materialized views for Apache Iceberg. Materialized views let you precompute an expensive query once and reuse the result across engines.

Run interactive workloads on Amazon EMR on EKS with Spark Connect

Amazon EMR on EKS now supports interactive Apache Spark sessions with Spark Connect. Data engineers and data scientists can develop and debug Apache Spark applications interactively from managed notebooks in Amazon SageMaker Unified Studio and their own IDEs, such as Jupyter and…

Amazon EMR on EKS now supports IPv6 Amazon EKS clusters

Today, AWS announces that Amazon EMR on EKS now supports running workloads on IPv6 Amazon Elastic Kubernetes Service (Amazon EKS) clusters.

Amazon EMR 7.14 is now available

with new features across Amazon EMR on EC2, Amazon EMR on EKS, and Amazon EMR Serverless, along with version upgrades for additional applications. This release also upgrades Apache Spark to 3.5.8 and Apache Iceberg to 1.10.1.

Amazon EMR introduces Long Term Support with Apache Spark 4.1

Amazon EMR introduces Long Term Support (LTS) releases, starting with emr-spark-8.1.0 and Apache Spark 4.1. With LTS, designated versions of the AWS runtime for Apache Spark receive 36 months of support.

Amazon EMR extends support at no additional cost for customers migrating off older releases

Amazon EMR now gives you more time, at no additional cost, to migrate off older releases before they reach end of support.

Amazon EMR on EKS now supports job run concurrency controls

Amazon EMR on EKS lets you run open-source big data frameworks such as Apache Spark and Flink on AWS EKS(Elastic Kubernetes Service) clusters.

Run interactive workloads on Amazon EMR on EC2 with Spark Connect

Amazon EMR on EC2 now supports interactive Apache Spark sessions with Spark Connect. Data engineers and data scientists can develop and debug Apache Spark applications interactively from managed notebooks in Amazon SageMaker Unified Studio and their own IDEs, such as Jupyter and…

Amazon EMR on EKS now supports Apache Spark troubleshooting agent

Amazon EMR on EKS now supports the Apache Spark troubleshooting agent. Data engineers can now diagnose EMR on EKS job failures through natural language, receiving automated root cause analysis and PySpark code recommendations without manually navigating distributed logs and…

Amazon EMR Serverless now supports larger worker sizes to run more compute and memory intensive workloads

Amazon EMR Serverless now offers larger worker configurations of 32 vCPUs with up to 244 GB of memory, allowing you to run more compute and memory-intensive workloads.

Amazon EMR Serverless now supports live configuration updates without application restarts

Amazon EMR Serverless now supports updates to key application configurations such as maximum capacity, and custom image settings — without stopping and restarting the application.

Amazon EMR now supports Apache Spark 4.0.2 in general availability

Amazon EMR now supports Apache Spark 4.0.2 across all three deployment models. With Spark 4.0.2, you can build and maintain data pipelines more easily with ANSI SQL and VARIANT data types, enforce fine-grained access control (FGAC) at the row level or column level, strengthen…

Run Interactive Workloads on Amazon EMR Serverless with Spark Connect

Amazon EMR Serverless now supports interactive sessions with Spark Connect, enabling you to develop and run Apache Spark applications from managed notebooks in Amazon SageMaker Unified Studio, as well as your favorite notebook environments and IDEs such as Jupyter and Visual…

Amazon EMR Serverless is now available in additional AWS Regions

Amazon EMR Serverless is now generally available in six additional AWS Regions - Asia Pacific (Hyderabad), Asia Pacific (Malaysia), Asia Pacific (New Zealand), Asia Pacific (Taipei), Asia Pacific (Thailand), and Mexico (Central).

Amazon EMR 7.13 now available with Python 3.11

Amazon EMR 7.13 is now available with Python 3.11 and version upgrades for additional applications.

Apache Spark troubleshooting and upgrade agents now available as Kiro powers

The Apache Spark troubleshooting agent and upgrade agent for Amazon EMR are now available as Kiro powers, bringing one-click access to AI-assisted Spark operations directly in Kiro.

Amazon EMR Serverless now supports AWS KMS customer managed keys for encrypting local disks

Amazon EMR Serverless now supports encrypting local disks with AWS Key Management Service (KMS) customer managed keys (CMKs).

Amazon EMR Serverless adds support for job run level cost allocation

Amazon EMR Serverless now supports job run-level cost allocation that provides better visibility into charges for individual job runs by allowing you to configure granular billing attribution at the individual job run level.