How to Monitor AWS Secrets Manager Rotation Failures: Step by Step Guide

When AWS Secrets Manager rotation fails silently, applications keep using old credentials until a database locks out the connection or an API rejects the request. By then, you are debugging in production under time pressure. A single rotation failure can cascade into authentication errors across multiple services if the secret is shared, and CloudWatch does not surface rotation state changes by default.

This guide walks through setting up CloudWatch alarms, EventBridge rules, and Lambda log monitoring to catch rotation failures the moment they happen. You will configure alerts for RotationFailed events, query Lambda rotation function logs for specific error patterns, and build a notification pipeline that routes failures to Slack or PagerDuty with full context.

Prerequisites

Before setting up rotation failure monitoring, verify you have the following in place:

• AWS account with Secrets Manager secrets already configured and rotation enabled
• IAM permissions to create CloudWatch alarms, EventBridge rules, SNS topics, and read CloudWatch Logs
• AWS CLI installed and configured, or access to the AWS Management Console
• A notification endpoint ready (SNS topic, Slack webhook, PagerDuty integration key, or email distribution list)
• At least one secret with rotation enabled and a Lambda rotation function attached
• Basic familiarity with CloudWatch Logs Insights query syntax if you plan to query Lambda logs programmatically

Step 1: Understand How Secrets Manager Rotation Events Work

AWS Secrets Manager publishes rotation status changes to CloudWatch Events (now part of EventBridge). When a rotation attempt completes, Secrets Manager emits an event with a detail-type of AWS API Call via CloudTrail or a service event depending on how the rotation was triggered.

The most reliable signal for rotation failure is the RotationFailed staging label. When rotation fails, Secrets Manager applies this label to the secret version that failed to rotate. You can detect this by monitoring for PutSecretValue API calls where the VersionStage includes AWSPENDING but the rotation never transitions to AWSCURRENT.

However, the cleaner approach is to monitor EventBridge for rotation completion events and check the rotation status directly. Secrets Manager does not publish a dedicated RotationFailed event type, but you can infer failure by monitoring for rotation initiation without a corresponding success event within the expected time window, or by parsing CloudWatch Logs from the Lambda rotation function.

For most teams, the fastest path to reliable alerting is to monitor the Lambda rotation function logs directly. Rotation failures surface as CloudWatch log entries with error messages like setSecret: Unable to log into database or ClientError: An error occurred (AccessDeniedException). These appear immediately when rotation fails, before any staging label logic runs.

Step 2: Create a CloudWatch Log Metric Filter for Rotation Errors

CloudWatch Logs Insights can query Lambda logs, but metric filters let you turn log patterns into numeric CloudWatch metrics that can trigger alarms.

Navigate to CloudWatch in the AWS Console, then go to Logs > Log groups. Find the log group for your Lambda rotation function. It will be named /aws/lambda/your-rotation-function-name. Click into the log group, then choose Actions > Create metric filter.

In the filter pattern field, enter a pattern that matches rotation errors. The exact pattern depends on your rotation function, but most AWS-managed rotation functions log errors with specific keywords. A general pattern that catches most failures:

[time, request_id, level=ERROR*, ...]

This matches any log entry where the level field contains ERROR. If your rotation function uses structured JSON logging, use a JSON filter pattern instead:

{ $.level = "ERROR" }

For the metric, assign it to a custom namespace like SecretsManager/Rotation. Name the metric RotationErrors and set the metric value to 1. Set the default value to 0 so periods without errors report zero instead of no data.

Click Create metric filter. This metric now increments by 1 every time the Lambda function logs an ERROR level message.

Step 3: Set Up a CloudWatch Alarm on the Rotation Error Metric

With the metric filter in place, create a CloudWatch alarm to trigger when rotation errors occur.

Go to CloudWatch > Alarms > Create alarm. Click Select metric, then browse to your custom namespace SecretsManager/Rotation and select the RotationErrors metric.

Set the alarm condition to trigger when the metric is greater than or equal to 1 for 1 consecutive period. Set the period to 5 minutes. This means the alarm fires if any rotation error appears in a 5 minute window.

For the alarm action, choose an existing SNS topic or create a new one. If you are setting up Slack or PagerDuty notifications, create an SNS topic now and configure subscriptions later. Name the topic something like secrets-rotation-failures.

Name the alarm SecretsManagerRotationFailure and add a description: “Triggered when AWS Secrets Manager rotation function logs an ERROR level message.”

Click Create alarm. Within 5 minutes of the next rotation failure, this alarm will fire and publish a notification to the SNS topic.

Step 4: Route Notifications to Slack or PagerDuty

SNS topic notifications default to email, but most teams need rotation failures routed to Slack channels or on-call tools like PagerDuty.

For Slack integration, create an AWS Lambda function that subscribes to the SNS topic and posts messages to a Slack webhook. Use the AWS Serverless Application Repository app cloudwatch-alarm-to-slack or write a simple Python Lambda:

import json
import urllib3

http = urllib3.PoolManager()

def lambda_handler(event, context):
    message = event['Records'][0]['Sns']['Message']
    alarm_data = json.loads(message)
    
    slack_message = {
        'text': f"🔴 Secrets Manager Rotation Failed",
        'attachments': [{
            'color': 'danger',
            'fields': [
                {'title': 'Alarm', 'value': alarm_data['AlarmName'], 'short': True},
                {'title': 'Reason', 'value': alarm_data['NewStateReason'], 'short': False}
            ]
        }]
    }
    
    encoded_msg = json.dumps(slack_message).encode('utf-8')
    resp = http.request('POST', 'YOUR_SLACK_WEBHOOK_URL', body=encoded_msg)
    
    return {'statusCode': 200}

Deploy this Lambda and subscribe it to the SNS topic. Grant the Lambda execution role permission to be invoked by SNS.

For PagerDuty, use the native AWS CloudWatch integration. Create an SNS topic subscription using your PagerDuty integration email address. PagerDuty will automatically create incidents when the SNS topic receives alarm notifications.

Step 5: Monitor Rotation Attempts with EventBridge Rules

CloudWatch alarms catch errors logged by the Lambda function, but they do not detect cases where rotation never starts or where the Lambda times out silently. To catch these, set up an EventBridge rule that monitors for rotation initiation without completion.

Go to EventBridge > Rules > Create rule. Name it SecretsManagerRotationMonitor. Under Event pattern, select AWS events, then choose Secrets Manager as the service.

Use this event pattern to capture rotation lifecycle events:

{
  "source": ["aws.secretsmanager"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": {
    "eventName": ["RotateSecret", "PutSecretValue"]
  }
}

This rule fires whenever a rotation starts (RotateSecret) or a secret version changes (PutSecretValue). For the target, choose a Lambda function that tracks rotation state in a DynamoDB table. When RotateSecret is called, store the secret ARN and timestamp. When PutSecretValue is called with AWSCURRENT, mark the rotation complete. If more than 30 minutes pass without completion, trigger an alert.

This approach requires more custom logic but catches failures that never generate error logs, like network timeouts or Lambda concurrency limits.

Step 6: Query Lambda Logs for Specific Error Patterns

CloudWatch Logs Insights lets you query rotation function logs to find patterns that metric filters miss. For example, if you need to detect setSecret: Unable to log into database errors specifically, use this query:

fields @timestamp, @message
| filter @message like /setSecret/
| filter @message like /Unable to log into database/
| sort @timestamp desc
| limit 20

Run this query against the /aws/lambda/your-rotation-function-name log group. It returns the 20 most recent log entries where the rotation function failed to connect to the database during the setSecret step.

To automate this, create a CloudWatch Logs Insights query and schedule it using a Lambda function triggered by EventBridge. If the query returns results, publish a notification to SNS. This pattern works for detecting specific failure modes that are too noisy to alarm on individually but matter in aggregate.

For teams using infrastructure monitoring tools beyond CloudWatch, exporting Lambda logs to a centralized log aggregator can simplify correlation with other signals. Platforms like CubeAPM ingest Lambda logs via OpenTelemetry and let you query across rotation failures, database errors, and network timeouts in a single interface without switching between AWS console tabs.

Step 7: Set Up Alerts for Secrets Nearing Expiration

Rotation failures often correlate with secrets approaching expiration. If rotation fails and the secret expires before the issue is fixed, applications lose access entirely.

AWS Secrets Manager does not natively publish expiration warnings to CloudWatch, but you can create a Lambda function that runs daily via EventBridge and checks secret metadata using the DescribeSecret API. If NextRotationDate is within 48 hours and rotation has not succeeded, trigger an alert.

Here is a minimal Python example:

import boto3
from datetime import datetime, timedelta

secrets_client = boto3.client('secretsmanager')
sns_client = boto3.client('sns')

def lambda_handler(event, context):
    secrets = secrets_client.list_secrets()
    
    for secret in secrets['SecretList']:
        if 'RotationEnabled' in secret and secret['RotationEnabled']:
            details = secrets_client.describe_secret(SecretId=secret['ARN'])
            next_rotation = details.get('NextRotationDate')
            
            if next_rotation and next_rotation < datetime.now() + timedelta(hours=48):
                sns_client.publish(
                    TopicArn='arn:aws:sns:region:account:secrets-rotation-failures',
                    Subject='Secret rotation approaching deadline',
                    Message=f"Secret {secret['Name']} rotates in less than 48 hours. Verify rotation is healthy."
                )

Schedule this Lambda to run every 24 hours. It scans all secrets with rotation enabled and alerts if any are approaching their next rotation window without recent successful rotation.

Troubleshooting Common Issues

Most rotation failures fall into a few categories. Understanding the error message speeds up resolution.

Error: Access to KMS is not allowed

The Lambda rotation function does not have permission to decrypt the secret using the KMS key that encrypted it. Check the KMS key policy and verify the Lambda execution role has kms:Decrypt permission. If the secret was encrypted with a customer managed key, the key policy must explicitly allow the Lambda role.

Error: setSecret: Unable to log into database

The rotation function successfully retrieved the new credentials but failed to connect to the database. Common causes: security group rules blocking Lambda egress, incorrect VPC configuration, database connection limits reached, or the new credentials were rejected because the database password policy was not met. Check CloudWatch logs for the exact connection error and verify the Lambda function runs in the same VPC as the database if required.

Error: Lambda timeout during rotation

Rotation functions have a default timeout of 30 seconds. If rotation involves multiple database round trips or slow network conditions, this is not enough. Increase the Lambda timeout to 60 or 90 seconds in the function configuration. If timeouts persist, check for network latency between Lambda and the database, or verify the database is not overloaded.

Error: A previous rotation is not complete. That rotation will be reattempted.

This appears when rotation is triggered while a previous attempt is still pending. Secrets Manager does not allow concurrent rotations for the same secret. Wait for the pending rotation to complete or fail before triggering a new one. If the pending rotation is stuck, manually remove the AWSPENDING staging label using the AWS CLI:

aws secretsmanager update-secret-version-stage \
  --secret-id your-secret-name \
  --version-stage AWSPENDING \
  --remove-from-version-id version-id-here

After removing the pending label, retry rotation.

No CloudWatch logs appear for rotation function

Verify the Lambda execution role has permission to write to CloudWatch Logs. The role needs logs:CreateLogGroup, logs:CreateLogStream, and logs:PutLogEvents permissions. If logs still do not appear, check the Lambda function configuration and confirm CloudWatch logging is not disabled.

For teams managing secrets across multiple AWS accounts or regions, correlating rotation failures with broader infrastructure health requires more than CloudWatch alone. AWS monitoring tools that aggregate logs, metrics, and traces across services help identify whether a rotation failure was caused by database overload, network partition, or application misconfiguration.

Conclusion

Monitoring AWS Secrets Manager rotation failures requires combining CloudWatch Logs metric filters, EventBridge rules, and proactive expiration checks. The most reliable approach is to monitor Lambda rotation function logs directly for ERROR level messages, create CloudWatch alarms that trigger on those patterns, and route notifications to Slack or PagerDuty. Adding EventBridge rules to track rotation lifecycle events catches cases where rotation never completes, and querying logs with CloudWatch Logs Insights helps surface specific error patterns that matter to your stack. Rotation failures that go undetected for hours turn into production incidents, so setting up alerts before the first failure happens is the only way to keep applications running when credentials rotate.

Disclaimer: The information in this article reflects the latest details available at the time of publication and may change as technologies and products evolve. Features, pricing, and plan limits can change over time. Always verify the latest information directly with the vendor before making purchasing or deployment decisions.

Frequently Asked Questions