AWS Lambda launches enhanced observability for Kafka event source mappings

AWS Lambda launches enhanced observability for Kafka event source mappings (ESM) that provides Amazon CloudWatch Logs and metrics to monitor event polling setup, scaling, and processing state of Kafka events. This capability allows customers to quickly diagnose setup issues and take timely corrective actions to operate resilient data streaming workloads. This capability is available for both Amazon Managed Streaming for Apache Kafka (Amazon MSK) and self-managed Apache Kafka (SMK) event source mappings.

Customers use Kafka event source mappings (ESM) with their Lambda functions to build mission-critical applications. However, the lack of visibility into event polling setup, scaling, and processing state for events slows down troubleshooting for issues resulting from faulty permissions, misconfiguration, or function errors, which increases mean time to resolution and adds operational overhead. With this launch, customers can enable CloudWatch Logs and metrics to monitor their Kafka polling setup, scaling, and event processing state. Customers can select from multiple log level options that provide logs ranging from warnings and errors to detailed information about event processing progress. Similarly, customers can enable one or more metrics groups—EventCount, ErrorCount, and KafkaMetrics—to monitor various aspects of event processing. Customers can view all their metrics and logs via a dedicated monitoring page on AWS Console for ESM. This capability allows customers to utilize their observability tooling to quickly diagnose setup issues and track performance metrics to meet their stringent business requirements.

This feature is available in all AWS Commercial Regions where AWS Lambda's Provisioned mode for Kafka ESM is available.

You can enable ESM logs and metrics for your Kafka ESM using AWS Lambda's Create and Update ESM APIs, AWS Console, AWS CLI, AWS SDK, AWS CloudFormation, and AWS SAM. To learn more about these capabilities, visit the Lambda Kafka ESM developer documentation. These logs and metrics are charged at standard CloudWatch pricing.