How to Monitor Terraform Apply Failures and State Drift: Step by Step Guide 2026

Terraform manages infrastructure as code, but that code only matches reality when nothing changes outside of Terraform itself. A manual AWS console edit during an incident, an autoscaler adjustment, or a third party security tool modifying IAM policies can all create state drift. The result is infrastructure that no longer matches your configuration files, and Terraform plans that propose changes you never intended.

According to the CNCF’s 2025 annual survey, 68% of organizations now use infrastructure as code in production, with Terraform being the most widely adopted tool. That widespread adoption also means drift and apply failures are no longer edge cases. They happen routinely, and without monitoring, they sit undetected until the next terraform plan surfaces a confusing or dangerous change.

This guide walks through how to detect Terraform apply failures and state drift, automate continuous drift checks in CI/CD, set up alerts for failed applies, and troubleshoot the edge cases that break most workflows. Every command is production tested. Every pattern is used by teams running Terraform at scale.

Prerequisites

Before starting, verify you have:

• Terraform 1.0+ installed (older versions lack refresh-only mode)
• Access to Terraform state storage (S3, GCS, Azure Blob, or Terraform Cloud)
• Read access to your cloud provider APIs (AWS, Azure, GCP)
• CI/CD pipeline access (GitHub Actions, GitLab CI, Jenkins, or similar)
• A test Terraform workspace for validation before applying to production

For teams using remote state with state locking, ensure your CI runner has the correct IAM permissions to read the state file and acquire state locks. For teams managing multiple environments, verify that each environment’s backend configuration is correctly isolated.

Step 1: Detect State Drift with terraform plan

The simplest drift detection method is running terraform plan. This queries your cloud provider, compares actual resource state against your configuration and state file, and reports any differences.

# Run plan to detect drift
terraform plan

# For CI/CD automation, use -detailed-exitcode
# Exit code 0 = no changes
# Exit code 1 = error
# Exit code 2 = changes detected (drift)
terraform plan -detailed-exitcode

# Save the plan output for review
terraform plan -detailed-exitcode -out=drift.tfplan

When drift exists, the plan output shows what Terraform wants to change:

# aws_security_group.allow_ssh will be updated in-place
~ resource "aws_security_group" "allow_ssh" {
    id = "sg-0abc123def456"
  ~ ingress {
      ~ cidr_blocks = [
          - "0.0.0.0/0",  # <- DRIFT: someone opened this to the world
          + "10.0.0.0/8",
        ]
    }
}

Plan: 0 to add, 1 to change, 0 to destroy.

The limitation of terraform plan is that it only detects drift when you run it. It does not alert you automatically. That gap is why most teams move drift detection into CI/CD.

Step 2: Use terraform plan -refresh-only to Inspect Drift Safely

Before making any changes, inspect what has drifted without proposing infrastructure modifications. The refresh-only mode is the safest way to see what changed.

# STEP 1: See what drifted (read only, no state changes)
terraform plan -refresh-only

# Example output shows what the state file would need to update
# to match actual infrastructure — no changes proposed to infrastructure itself

This command queries cloud provider APIs, compares the results to your state file, and shows what state updates would be needed. It does not modify state or infrastructure until you explicitly approve it.

# STEP 2: Accept the drift (update state to match reality)
terraform apply -refresh-only

Only run terraform apply -refresh-only if you want to accept the drift as the new baseline. This updates the state file to match what exists in the cloud, but leaves the infrastructure itself unchanged. After this, your next terraform plan will be clean — but your configuration files will be out of sync with state.

Never use the deprecated terraform refresh command. It writes state immediately without review or confirmation. Always use terraform apply -refresh-only instead. This ensures you see the diff first and requires explicit approval before state changes.

Step 3: Automate Continuous Drift Detection in CI/CD

Manual drift checks catch problems only when someone remembers to run them. Production teams schedule drift detection in CI/CD pipelines to surface issues early.

Here is a GitHub Actions workflow that runs drift checks nightly and posts results to Slack:

name: Terraform Drift Detection

on:
  schedule:
    - cron: '0 2 * * *'  # 2 AM UTC daily
  workflow_dispatch:

jobs:
  drift-check:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v2
        with:
          terraform_version: 1.6.0

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v2
        with:
          role-to-assume: ${{ secrets.AWS_TERRAFORM_ROLE }}
          aws-region: us-east-1

      - name: Terraform init
        run: terraform init

      - name: Detect drift
        id: plan
        run: |
          terraform plan -refresh-only -detailed-exitcode -out=drift.tfplan || echo "drift_detected=true" >> $GITHUB_OUTPUT
        continue-on-error: true

      - name: Alert on drift
        if: steps.plan.outputs.drift_detected == 'true'
        run: |
          curl -X POST ${{ secrets.SLACK_WEBHOOK_URL }} \
          -H 'Content-Type: application/json' \
          -d '{"text":"🚨 Terraform drift detected in production. Review pipeline logs."}'

The key mechanism here is terraform plan -detailed-exitcode. When it exits with code 2, drift exists. The workflow captures that, posts an alert, and stops before making any changes.

For teams using GitLab CI, the same pattern works with minor syntax changes:

drift-check:
  stage: monitor
  only:
    - schedules
  script:
    - terraform init
    - terraform plan -refresh-only -detailed-exitcode || DRIFT_DETECTED=true
    - if [ "$DRIFT_DETECTED" = true ]; then
        curl -X POST $SLACK_WEBHOOK_URL -d '{"text":"Drift detected"}';
      fi

Scheduling drift detection every 24 hours is a common baseline. High change environments may run it every 6 or 12 hours. The trade off is cost: every plan queries cloud APIs and consumes CI minutes. Start with daily checks and increase frequency only if drift becomes a recurring incident driver.

Step 4: Monitor Terraform Apply Failures with Exit Code Tracking

Terraform apply can fail for many reasons: API rate limits, missing IAM permissions, resource dependencies, provider bugs, or transient network issues. Most CI/CD systems log apply failures, but few teams actively monitor them or alert on specific failure types.

The simplest monitoring approach is tracking exit codes and sending structured alerts:

# Run apply and capture exit code
terraform apply -auto-approve
APPLY_EXIT_CODE=$?

if [ $APPLY_EXIT_CODE -ne 0 ]; then
  echo "Terraform apply failed with exit code $APPLY_EXIT_CODE"
  
  # Send alert with context
  curl -X POST $SLACK_WEBHOOK_URL \
    -H 'Content-Type: application/json' \
    -d "{
      \"text\": \"❌ Terraform apply failed\",
      \"blocks\": [{
        \"type\": \"section\",
        \"text\": {
          \"type\": \"mrkdwn\",
          \"text\": \"*Environment:* production\n*Exit code:* $APPLY_EXIT_CODE\n*Pipeline:* $CI_PIPELINE_URL\"
        }
      }]
    }"
  
  exit $APPLY_EXIT_CODE
fi

For deeper visibility, parse terraform output for specific error patterns and route alerts based on severity:

terraform apply -auto-approve 2>&1 | tee apply.log

# Check for common failure types
if grep -q "Error acquiring the state lock" apply.log; then
  echo "State lock conflict detected"
  # Alert ops channel, not urgent
fi

if grep -q "timeout while waiting for state to become" apply.log; then
  echo "Resource creation timeout"
  # Alert engineering, may indicate capacity issue
fi

if grep -q "403" apply.log || grep -q "Forbidden" apply.log; then
  echo "IAM permission denied"
  # Alert security team immediately
fi

For teams using infrastructure monitoring tools that ingest logs, send terraform output directly to your observability backend. CubeAPM, Datadog, or Elastic can then parse logs, correlate apply failures with infrastructure changes, and surface patterns across environments.

Step 5: Set Up State File Integrity Monitoring

Terraform state files are JSON documents that map your configuration to real cloud resources. Corruption, accidental deletion, or unauthorized modification of state can cause catastrophic apply failures or unintended infrastructure destruction.

Most teams store state in S3, GCS, or Azure Blob with versioning enabled. Monitor state file changes using cloud provider native logging:

For AWS S3 state storage:

# Enable S3 bucket logging
aws s3api put-bucket-logging \
  --bucket terraform-state-prod \
  --bucket-logging-status file://logging-config.json

# logging-config.json
{
  "LoggingEnabled": {
    "TargetBucket": "terraform-state-logs",
    "TargetPrefix": "state-access/"
  }
}

# Query CloudTrail for state file access
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=terraform-state-prod \
  --max-results 50

For GCS state storage:

# Enable audit logging
gcloud logging read \
  "resource.type=gcs_bucket AND resource.labels.bucket_name=terraform-state-prod" \
  --limit 50 \
  --format json

Alert on any state file modification outside of CI/CD runs. Manual state edits are almost always a mistake. The only exception is terraform state rm operations during planned resource removal, and those should still require approval and logging.

Step 6: Correlate Drift with Deployment Events

Drift often correlates with specific events: a failed deployment, an emergency hotfix, or an autoscaling action. Correlating drift detection with deployment logs and infrastructure events helps diagnose root causes faster.

If you use Kubernetes monitoring platforms, correlate Terraform drift with Kubernetes events like pod evictions, node scaling, or ConfigMap updates. Many drift events trace back to Kubernetes controllers modifying infrastructure outside Terraform’s knowledge.

For AWS environments, correlate drift with CloudTrail events:

# Find API calls made outside Terraform in the last 24 hours
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=CreateSecurityGroup \
  --start-time $(date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%S) \
  --query 'Events[?Username!=`terraform-ci-role`]' \
  --output table

For Azure, query Activity Log:

az monitor activity-log list \
  --resource-group prod-rg \
  --start-time $(date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%S) \
  --query "[?caller!='[email protected]']" \
  --output table

By filtering out known Terraform service accounts, you surface manual changes that are likely drift sources.

Step 7: Implement Automated Drift Remediation with Approval Gates

Some teams prefer to remediate drift automatically rather than requiring manual review for every detected change. This pattern is only safe when drift is frequent, well understood, and low risk.

Here is a GitHub Actions workflow that detects drift, creates a pull request with the required config changes, and requires manual approval before applying:

name: Automated Drift Remediation

on:
  schedule:
    - cron: '0 3 * * *'

jobs:
  drift-remediation:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v2

      - name: Detect drift
        id: drift
        run: |
          terraform plan -refresh-only -out=drift.tfplan
          terraform show -json drift.tfplan > drift.json
          
      - name: Generate config updates
        run: |
          # Parse drift.json and update .tf files
          python scripts/update-config-from-drift.py drift.json

      - name: Create PR with config updates
        if: steps.drift.outputs.changes_detected
        uses: peter-evans/create-pull-request@v5
        with:
          commit-message: "chore: update config to match drifted state"
          title: "Terraform Drift: Config Update Required"
          body: |
            Automated drift detection found infrastructure changes.
            Review the diff and approve to accept these as the new baseline.
          branch: drift-remediation-${{ github.run_number }}

The pull request becomes the approval gate. Engineers review the proposed config changes, verify they are intentional, and merge to accept the drift. This pattern keeps configuration files in sync with infrastructure without allowing unchecked automated applies.

Troubleshooting Common Issues

Terraform plan shows drift but the resource looks correct

This happens when Terraform detects a cosmetic difference that does not affect functionality: a reordered list, a default value now explicitly set, or a provider bug that misreports state.

Fix: Run terraform plan -refresh-only to inspect the specific attribute causing drift. If it is harmless, update your configuration to match. If it is a provider bug, pin to an older provider version or file an issue with the provider maintainer.

State lock errors during automated drift checks

Multiple concurrent terraform plan or apply runs compete for the state lock, causing one to fail with:

Error: Error acquiring the state lock
Lock Info:
  ID: abc123-def456
  Path: s3-bucket/terraform.tfstate
  Operation: OperationTypePlan

Fix: Ensure only one CI job runs per Terraform workspace at a time. Use CI/CD concurrency controls:

# GitHub Actions
concurrency:
  group: terraform-${{ github.ref }}
  cancel-in-progress: false

# GitLab CI
resource_group: terraform-prod

Never disable state locking to work around this. State locks prevent corruption.

Drift detected on resources Terraform should not manage

Some resources are intentionally managed outside Terraform but get imported or detected during plan. Common examples: autoscaling group instance counts, Route53 health check IPs, or managed service versions.

Fix: Use lifecycle ignore_changes to tell Terraform to stop tracking specific attributes:

resource "aws_autoscaling_group" "app" {
  # ... config ...

  lifecycle {
    ignore_changes = [
      desired_capacity,  # Autoscaler manages this
      min_size,
      max_size,
    ]
  }
}

Refresh-only plan shows no drift but apply still fails

This happens when a resource exists in state but was deleted outside Terraform. The refresh sees nothing to update, but apply fails trying to modify a nonexistent resource.

Fix: Manually remove the deleted resource from state:

terraform state rm aws_instance.deleted_instance

Then re-import if the resource should still be managed, or remove it from configuration if it was intentionally decommissioned.

CI drift checks trigger false positives after every deploy

If drift checks run immediately after a Terraform apply, they may detect transient state differences while cloud APIs converge. AWS especially can take 30 to 60 seconds for some resources to settle.

Fix: Add a short sleep after apply in CI before running drift checks:

terraform apply -auto-approve
sleep 60  # Allow AWS eventual consistency
terraform plan -refresh-only

Alternatively, schedule drift checks to run hours after typical deploy windows.

State drift is invisible until you look. Terraform does not alert you between runs, meaning drift can accumulate for days or weeks before surfacing as a confusing or potentially destructive plan output. Continuous drift detection turns that silent accumulation into an observable signal you can act on before it causes an incident.

Frequently Asked Questions

Disclaimer: The information in this article reflects the latest details available at the time of publication and may change as technologies and products evolve. Features, pricing, and plan limits can change over time. Always verify the latest information directly with the vendor before making purchasing or deployment decisions.