CloudWatch Evidently: Feature Flag and A/B Test Monitoring

AWS CloudWatch Evidently was a managed service for feature flags, A/B testing, and experimentation launched in late 2021. It allowed teams to roll out features to specific user segments, measure performance impact through custom metrics, and make deployment decisions based on real user data rather than guesswork. On October 16, 2025, AWS discontinued Evidently entirely and recommended migration to AWS AppConfig feature flags as the replacement path.

This guide covers what Evidently was, how it worked, why AWS deprecated it, and what alternatives exist today for teams that need feature flags, A/B testing, and deployment experimentation with observability built in.

What Was CloudWatch Evidently?

CloudWatch Evidently was AWS’s managed feature experimentation platform. It provided three core capabilities: feature flags for controlled rollouts, launches for gradual traffic splitting, and experiments for statistically valid A/B testing with business metric tracking.

Feature flags in Evidently allowed teams to toggle features on or off for specific user segments without deploying new code. Launches enabled percentage based traffic splits, rolling out a new checkout flow to 10% of users while keeping 90% on the existing experience. Experiments tied those rollouts to custom metrics like conversion rate or page load time and applied statistical analysis to determine which variant performed better.

Evidently integrated directly with CloudWatch Logs and S3 for storing evaluation events. Teams could send performance data using the putProjectEvents API and Evidently would aggregate it, calculate confidence intervals, and surface recommendations on which variant won. This made it possible to answer questions like “did the new feature increase sign ups?” or “did latency get worse after the change?” with statistical confidence rather than intuition.

The service used a project based structure. Each project contained multiple features, and each feature could have multiple variations (true/false for boolean flags, or string/number values for configuration). Launches mapped variations to user segments with traffic weights. Experiments ran on top of launches and measured the impact of each variation against defined metric goals.

How CloudWatch Evidently Worked

Evidently’s architecture had four layers: projects, features, launches, and experiments. Each served a specific role in the deployment and measurement pipeline.

Projects and Data Delivery

An Evidently project was the top level container. It defined where evaluation events were stored, either CloudWatch Logs or S3. Evaluation events recorded every time a feature flag was checked by application code, capturing the entity ID (usually a user ID), the feature name, the variation returned, and a timestamp. These events formed the dataset that launches and experiments analyzed.

Setting up data delivery to S3 allowed long term retention and integration with data lakes. CloudWatch Logs integration enabled real time log analysis and alerting. Both options required explicit IAM permissions for Evidently to write to the destination.

Features and Variations

A feature represented a single capability being controlled. It had a name, a description, and a set of variations. For a boolean feature flag, variations were typically enabled (true) and disabled (false). For configuration flags, variations could be strings, numbers, or more complex JSON structures.

Each feature defined a default variation. If no launch or experiment applied to a given user, the default variation was returned. This ensured consistent behavior for users outside any active rollout or test.

Launches for Controlled Rollouts

Launches split traffic between variations. A launch defined groups (called treatment groups), each mapped to a specific variation of a feature. Each group had a traffic weight, expressed as an integer between 0 and 100,000 (representing 0% to 100% with precision to 0.001%).

For example, a launch rolling out a new UI to 20% of users would define two groups: control-group at 80,000 (80%) mapped to the disabled variation, and treatment-group at 20,000 (20%) mapped to the enabled variation. The split happened at evaluation time based on a hash of the entity ID, ensuring consistent user assignment across sessions.

Launches could be scheduled. A scheduledSplitsConfig allowed teams to define traffic ramps over time, starting a feature at 5% on Monday, increasing to 25% on Wednesday, and reaching 100% by Friday if no issues emerged.

Experiments and Metric Goals

Experiments added statistical rigor. An experiment defined metric goals tied to business outcomes. Each goal had a metricName, a desiredChange (INCREASE or DECREASE), and an eventPattern that filtered which evaluation events contributed to the metric.

For example, an experiment measuring page load time would define a metric goal with desiredChange: DECREASE, valueKey: details.loadTime, and an event pattern matching only page load events. Application code sent custom events using putProjectEvents, including the entity ID, event type, and metric value. Evidently aggregated these, calculated mean performance per variation, and ran statistical tests to determine if differences were significant.

Experiments ran alongside launches. The launch controlled traffic distribution; the experiment measured the outcome. This separation allowed teams to run experiments on a small percentage of traffic before making a full rollout decision.

Why AWS Deprecated CloudWatch Evidently

AWS announced Evidently’s deprecation in April 2025 and shut it down on October 16, 2025. The official reason was strategic consolidation. AWS already had AWS AppConfig, a separate service for feature flags and configuration management. Maintaining two overlapping services created confusion and split engineering investment.

AppConfig offered simpler feature flag management without the experiment layer. AWS positioned it as the long term path for feature flags, leaving experimentation to third party tools or custom implementations on top of AppConfig. This meant Evidently’s statistical A/B testing capability had no direct AWS replacement, teams needing that functionality had to move to external platforms like LaunchDarkly, Split, or Optimizely.

The deprecation timeline gave teams six months to migrate. AWS provided migration guides to AppConfig but offered no automated tooling. Teams had to manually recreate feature flags, rewrite evaluation logic to use AppConfig APIs, and rebuild any experiment frameworks from scratch.

According to discussions on Reddit and AWS forums, many teams saw the deprecation as a sign that AWS was not committed to the experimentation space. Some moved to AppConfig for flags only. Others chose third party platforms that bundled flags and experiments together. A smaller group built custom solutions on top of open source frameworks.

AWS AppConfig: The Recommended Migration Path

AWS AppConfig became the official replacement for Evidently feature flags. It provides a simpler model focused on configuration management rather than experimentation.

How AppConfig Differs from Evidently

AppConfig does not include experiments, statistical analysis, or metric goal tracking. It is purely a feature flag and configuration service. You define a flag, set its value, and deploy it to an environment. There is no built in A/B test framework.

AppConfig uses a different evaluation model. Instead of sending user IDs to AWS and getting a variation back, AppConfig delivers the entire configuration to the client application. The client evaluates rules locally. This reduces latency but shifts complexity to the client side.

AppConfig supports percentage based rollouts and targeting rules. You can roll out a flag to 15% of users or target specific user attributes (like region or account tier). But there is no automatic statistical evaluation of whether the rollout improved a business metric.

Migration Steps from Evidently to AppConfig

Migrating from Evidently to AppConfig required three major changes: recreating feature definitions, replacing evaluation API calls, and rebuilding any experiment tracking outside AWS.

First, each Evidently feature had to be recreated as an AppConfig feature flag. AppConfig flags use a JSON schema to define variations and targeting rules. A boolean flag in Evidently became an AppConfig flag with enabled: true and enabled: false variations.

Second, application code had to replace evaluateFeature API calls with AppConfig’s client side evaluation. This meant integrating the AWS AppConfig Agent or SDKs, fetching the configuration on startup, and checking flag values locally. The shift from server side to client side evaluation introduced caching complexity and required handling configuration updates without restarting the application.

Third, any experiment logic had to be rebuilt. Teams relying on Evidently’s putProjectEvents API and statistical analysis had to either switch to a third party experimentation platform or build custom metric collection and analysis pipelines. Most teams chose third party tools because building robust statistical testing from scratch is complex and error prone.

Feature Flags and Experimentation After Evidently

Evidently’s deprecation left a gap for teams that wanted managed experimentation tightly integrated with AWS infrastructure. Several alternatives emerged, each with different trade offs on cost, deployment model, and feature depth.

Managed Experimentation Platforms

LaunchDarkly, Split, and Optimizely are the leading SaaS platforms for feature flags and experimentation. They provide statistical A/B testing, metric tracking, and advanced targeting rules. Pricing is typically user based or event based, which can scale unpredictably as traffic grows.

LaunchDarkly starts around $10 per seat per month for the Starter plan and scales into custom enterprise pricing. Split uses a similar model. Both platforms integrate with CloudWatch and other infrastructure monitoring tools for pulling in operational metrics alongside business metrics.

The main advantage of these platforms is maturity. They have years of development behind statistical engines, sophisticated targeting, and built in integrations. The main downside is cost and data egress. Sending feature evaluation events to a third party SaaS means telemetry data leaves your AWS environment, which can introduce compliance concerns and AWS data transfer fees.

Self Hosted Feature Flag Platforms

Unleash and Flagsmith are open source feature flag platforms that can run inside your own AWS VPC. Unleash provides a full featured flag management UI, SDKs for 15+ languages, and basic experimentation through variant tracking. Flagsmith offers similar capabilities with a different UI and pricing model.

Running these platforms self hosted means no per seat fees and full data control. The trade off is operational overhead. You manage the database, handle upgrades, and monitor the flag service itself. For teams already running Kubernetes clusters and comfortable with operational complexity, this is often the preferred path.

Observability Platforms with Built-In Feature Flag Support

Some observability platforms now include feature flag evaluation and experiment tracking as part of their core offering. This eliminates the need for a separate feature flag service and unifies deployment decisions with performance monitoring.

CubeAPM provides feature flag evaluation and A/B test tracking as part of its unified observability platform. It runs entirely inside your own cloud or data center, so no telemetry data leaves your infrastructure. Feature flag evaluations are logged alongside traces, metrics, and logs, making it easier to correlate deployment changes with performance regressions. Because CubeAPM uses a flat $0.15/GB pricing model with no per user or per event fees, the cost remains predictable even as traffic scales.

This approach works well for teams that want observability and feature management in one platform rather than integrating multiple SaaS tools. It is particularly relevant for teams with data residency requirements or those migrating away from multi tool SaaS sprawl.

Monitoring Feature Flag Impact Without Evidently

Even without Evidently’s built in experiment framework, teams still need to measure how feature rollouts affect application performance and business outcomes. The core requirement is correlating feature flag state with telemetry data.

Tagging Telemetry with Feature Flag Context

The simplest method is tagging every trace, log, and metric with the active feature flags for that request. OpenTelemetry supports adding custom attributes to spans. When evaluating a feature flag, the application records the flag name and variant as span attributes. Observability tools that ingest OpenTelemetry data can then filter and compare performance metrics grouped by flag state.

For example, if a new database query optimization is behind a feature flag, traces tagged with feature.db_optimization=enabled can be compared to traces with feature.db_optimization=disabled to measure latency differences.

Statistical Analysis Outside the Feature Flag Service

Without Evidently’s automatic statistical testing, teams must export metric data and run analysis separately. Options include exporting CloudWatch metrics to S3 and running analysis in Jupyter notebooks, streaming logs to a data warehouse and using SQL for cohort comparisons, or integrating with business intelligence tools like Looker or Tableau.

For small experiments, manual comparison is sufficient. For larger scale continuous experimentation, third party platforms or custom tooling become necessary.

Best Practices for Feature Flags and A/B Testing Post Evidently

Teams migrating from Evidently or building new experimentation workflows should follow a few structural best practices to avoid common pitfalls.

Keep Feature Flag Lifecycles Short

Feature flags should be temporary by default. A flag that remains in code for months becomes technical debt. After a feature has been fully rolled out or rolled back, the flag and all conditional code paths should be removed. Establishing a policy that flags have a maximum lifespan (30, 60, or 90 days) helps prevent flag sprawl.

Use Consistent Entity IDs for User Assignment

Traffic splitting and A/B tests depend on consistent user assignment. The entity ID used for flag evaluation must be stable across sessions. Using session IDs instead of user IDs will cause the same user to see different experiences on different visits, invalidating experiment results.

For anonymous users, a stable cookie based identifier or device fingerprint works. For authenticated users, the account ID is the correct choice.

Monitor Flag Evaluation Latency

Feature flag evaluation adds latency to every request. Server side evaluation incurs a network round trip. Client side evaluation requires configuration fetches and local caching. Monitoring the P95 and P99 latency of flag evaluation helps catch performance regressions before they degrade user experience.

Decouple Deployment from Release

Feature flags decouple code deployment from feature release. Code can be deployed to production with new features hidden behind flags. Once the code is stable, the flag is toggled to expose the feature. This separation reduces deployment risk and makes rollbacks instant.

Tools and Implementation

Modern feature flag and experimentation workflows require integrating multiple tools. The core stack includes a feature flag service, observability platform, and optional experimentation platform.

Feature Flag Service Options

AWS AppConfig for simple flags with native AWS integration, Unleash or Flagsmith for self hosted open source solutions, LaunchDarkly or Split for managed SaaS with built in experimentation, or CubeAPM for unified observability and feature flag evaluation in one self hosted platform.

Observability Platform Integration

Feature flag decisions must be visible in observability telemetry. Platforms like CubeAPM natively integrate flag state with traces and logs, eliminating the need for manual tagging. For teams using separate tools, adding flag state as OpenTelemetry span attributes is the standard approach.

Synthetic monitoring tools can also verify that flag enabled features work correctly in production by running automated tests against specific flag states.

Experiment Analysis Tools

For teams running statistical A/B tests without a built in platform, exporting data to Jupyter notebooks, using SQL based analysis in data warehouses like Snowflake or BigQuery, or integrating with business intelligence dashboards are the most common approaches.

Conclusion

CloudWatch Evidently provided a simple path to feature flags and experimentation directly within AWS, but its deprecation in October 2025 forced teams to rebuild those workflows. AWS AppConfig replaced Evidently for feature flags but removed the experimentation layer entirely, leaving teams to choose between third party SaaS platforms, self hosted open source tools, or building custom solutions.

For teams prioritizing cost predictability and data control, self hosted platforms like CubeAPM offer a unified path forward, combining feature flag management with full stack observability in one deployment. For teams willing to pay for managed services, LaunchDarkly and Split provide mature experimentation platforms with statistical rigor built in.

The most important lesson from Evidently’s deprecation is that feature flag tooling must be evaluated not just on features but on longevity, cost model, and integration complexity. Teams that built deeply on Evidently faced significant migration costs. Choosing tools with open APIs, standard telemetry formats like OpenTelemetry, and strong community or vendor commitment reduces that risk in future transitions.

Disclaimer: The information in this article reflects the latest details available at the time of publication and may change as technologies and products evolve. Features, pricing, and plan limits can change over time. Always verify the latest information directly with the vendor before making purchasing or deployment decisions.

Frequently Asked Questions