Data integration (ETL)

Amazon EMR now supports native data quality checks in Apache Spark jobs

Amazon EMR now validates data quality natively within Apache Spark jobs. Data engineering teams can catch quality issues inline before bad data reaches downstream systems, without building or maintaining separate validation infrastructure.

AWS Glue Data Catalog now supports table optimization, statistics, and crawlers for Apache Iceberg V3

AWS Glue Data Catalog now supports table optimization, statistics, and crawlers for Apache Iceberg Version 3 (V3) tables. With these new capabilities, you can automatically maintain V3 tables, optimize them for query performance, and discover them in Amazon S3.

Apache Iceberg materialized views now support system-managed write protection

Today, AWS announces system-managed materialized views for Apache Iceberg. Materialized views let you precompute an expensive query once and reuse the result across engines.

AWS Glue Data Quality delivers context-specific rule recommendations in seconds

AWS Glue Data Quality now generates data quality rules in seconds, reducing the time to establish data quality checks for your tables in the AWS Glue Data Catalog.

The BigQuery Source plugin version 0.24.7 is available in Cloud Data Fusion version 6.11.0 and later.

This release includes the following change:

AWS Glue zero-ETL now captures archived Salesforce records

AWS Glue zero-ETL integrations for Salesforce now capture archived records in your target tables, giving you a complete replica of your Salesforce data for analytics and reporting.

AWS Glue zero-ETL adds target table property ownership and conflict detection

AWS Glue zero-ETL integrations now detect table property conflicts and track integration ownership.

AWS Glue 5.1 is now available in AWS European Sovereign Cloud Region

AWS Glue 5.1 is now available in the AWS European Sovereign Cloud Region.

AWS Glue now supports catalog federation for remote Apache Iceberg catalogs in AWS GovCloud (US) regions

AWS Glue now supports catalog federation for remote Iceberg REST catalogs in AWS GovCloud (US) Regions. This capability provides direct and secure access to Iceberg tables, stored in Amazon S3 and cataloged in remote catalogs, using AWS analytics engines.

AWS Glue 6.0 delivers 30% price reduction and Iceberg v3 support

AWS Glue 6.0 is now generally available, delivering a 30% price reduction and introducing full support for Apache Iceberg v3, newer versions of Apache Hudi and Delta Lake, and new capabilities to improve developer productivity.

AWS Glue Schema Registry is now available in ten more AWS regions

You can now use the AWS Glue Schema Registry, a serverless and free feature of AWS Glue, in the Asia Pacific (New Zealand), Asia Pacific (Thailand), Asia Pacific (Hyderabad), Asia Pacific (Osaka), Asia Pacific (Malaysia), Asia Pacific (Melbourne), Mexico (Central), Israel (Tel…

AWS Glue Data Catalog now supports metadata exports to S3 Tables (Preview)

AWS Glue Data Catalog is a centralized metadata repository that helps you discover and understand data, with semantic search and support for adding business context to datasets currently in preview.

AWS Glue Data Quality makes ETL anomaly detection free and improves anomaly predictions

AWS Glue Data Quality now offers improved anomaly detection with a new observation mode that reduces detection of false anomalies and removes pricing for anomaly detection in ETL jobs.

AWS Glue announces VPC support, filter pushdown, and partition support for the REST API connector

AWS Glue now supports VPC connections, filter pushdown, and partition support for the REST API connector.

AWS Glue Data Quality now supports anomaly detection and writing results to the AWS Glue Data Catalog

AWS Glue Data Quality now supports anomaly detection for Catalog-based data quality evaluations and the ability to write evaluation results to AWS Glue Data Catalog (GDC) tables.

AWS Glue Data Quality now supports distribution statistics for data profiling

AWS Glue Data Quality now supports a new Distribution Analyzer that generates frequency distribution profiles for your data.

AWS Glue SAP OData connector and zero-ETL integrations are now available in AWS GovCloud (US) regions

AWS Glue SAP OData connector and zero-ETL integrations are now available in AWS GovCloud (US-West) and AWS GovCloud (US-East) Regions, with support for Amazon DynamoDB, Salesforce, and SAP OData as sources.

AWS Glue Data Catalog now supports business context and semantic search (Preview)

Today, AWS announces the preview of business context and semantic search for AWS Glue Data Catalog, helping you discover and understand data by semantic meaning.

AWS Glue Interactive Sessions now support Spark Connect for interactive workloads

AWS Glue Interactive Sessions now support Apache Spark Connect, using which you can now develop and run Apache Spark applications from your preferred environment, including managed notebooks in Amazon SageMaker Unified Studio, or your preferred notebook environments and IDEs…

Simplified permissions for Amazon S3 Tables and Iceberg materialized views are now available in AWS GovCloud (US) Regions

AWS Glue Data Catalog now supports AWS IAM-based authorization for Amazon S3 Tables and Apache Iceberg materialized views. With IAM-based authorization, you can define all necessary permissions across storage, catalog, and query engines in a single IAM policy.