# Use Case: Release Verification, One Flow Checked Before the Deploy and Monitored After It

Dokimore Support Team · October 28, 2026, updated 2026-11-01 · 8 min read

Format: Scenario walkthrough · Written for: QA leads and test managers, SDETs and automation engineers, Engineering managers · Where it applies: every market

> A test that passes on staging proves the build. A monitor that passes in production proves the release. They can check the same flow.

## In short

- Pre-deploy tests answer "does this build work on an environment we control"; production monitors answer "does the release work where the customers are". A team that only has the first learns about production-only failures, a missing environment variable, a dependency returning errors, a rotated certificate, from support tickets.
- The SREcon24 EMEA argument from Carly Richmond of Elastic is that the two are two sides of the same coin and should share tooling: one definition of the user journey, validated before the release and run as a monitor in production.
- In Dokimore the flow is the shared definition: its recording replays as the regression test before the deploy, and the API collection behind the same flow runs on a schedule as a monitor after it, with alerts to Slack, Microsoft Teams, Opsgenie or a webhook. There is no CI trigger yet; the exported code is what runs in a pipeline.

This is a scenario walkthrough of a release-verification setup. It is not a customer story, and the team is invented so the sequence is concrete; the outside facts are in the sources at the end.

## The scenario

A team deploys a SaaS application eight times a week. Its end-to-end suite runs against staging on every deploy and has been green for a month. In the same month, customers reported three failures first: a sign-in that broke for one identity provider after a certificate rotated, a search page that returned nothing because a production-only environment variable was missing, and an afternoon when a third-party payment service answered errors and the checkout showed a spinner. None of the three could have failed on staging, because none of the three existed on staging.

## Two questions, two environments

A pre-deploy test answers one question: does this build do what the product is supposed to do, on an environment we control. A production monitor answers a different one: does the release still do it, where the customers are, with the real certificates, the real configuration and the real dependencies. The first catches regressions in the code; the second catches regressions in everything else. A team that has only the first is not wrong, it is half-covered, and the half it lacks is the half that pages people at night.

Carly Richmond of Elastic made this case at SREcon24 EMEA in October 2024 under the title "Synthetic Monitoring and E2E Testing: 2 Sides of the Same Coin": teams use different tools for similar validation, in production with synthetic monitoring and in development with end-to-end testing, and could share one definition of the user journey across both. The example given was Playwright, GitHub Actions and Elastic Synthetics; the principle does not depend on the tools.

The strongest recent example of a failure only production can show is Cloudflare’s outage of 18 November 2025. A permissions change in a database system doubled the size of a configuration file for its Bot Management product; the file crossed a 200-feature limit in the software reading it, and the network returned errors from 11:20 UTC until core traffic recovered at 14:30. Nothing in any customer’s staging environment changed. A production monitor on the customer’s own checkout would have failed on its next scheduled run; a staging suite would have stayed green all afternoon.

## One definition, two jobs

In the scenario, the team records its six money journeys once, in a real browser, against a test account: sign-in with each identity provider, search, the checkout, the account page, the export. Each recording keeps the element and locators per step, the network calls each step caused and a screenshot, and replays as a regression test straight away.

Before a deploy, the folder of six recordings runs against staging, on demand from the release checklist or on the schedule the team sets. Scheduled folder runs use Chromium; the cross-browser replays run on the execution grid, which the Team plan includes. The run detail shows pass or fail per step with the screenshot, which is what the release checklist reads.

After the deploy, the API collection behind the same journeys, the search endpoint, the checkout calls and a request that depends on the payment provider, a test-mode charge or its status endpoint, runs on a schedule as a monitor against production, with assertions on status, body and timing. A failing assertion sends an alert to Slack, Microsoft Teams, Opsgenie or a webhook. A single-sign-on redirect is a browser journey that an API call does not exercise, so the sign-in recordings are recorded once more against production, with a test account for each identity provider, and run on their own schedule.

In the scenario, the missing environment variable fails the search assertion on the first monitor run after the deploy; the payment provider’s afternoon of errors fails the payment request’s assertion while it is happening; and the rotated certificate fails the scheduled sign-in recording at the identity provider’s step, with the screenshot of what a customer would have seen.

## What to assert, and what to leave out of a production monitor

- Assert outcomes, not timing, in the gate: a slower staging environment should not fail a working feature. Keep timing assertions for the monitor, where the environment is the one that matters.
- Keep the production monitor read-mostly. A monitor that creates orders needs a test account, synthetic data and a cleanup; a monitor that signs in, searches and reads the cart needs none of that and still catches the three failures in the scenario.
- Treat a failing monitor differently from a failing gate. A failing gate blocks a branch; a failing monitor opens an incident. The same assertion can mean a different action in the two places.
- Watch the third party on purpose. The checkout monitor includes a request that depends on the payment provider, so an afternoon of errors there is seen by the monitor before it is seen by a customer.

## What runs where, today

The pre-deploy run happens in Dokimore, on demand or on a schedule; there is no CI trigger, CI token or quality gate from the platform yet. A team that needs the gate inside its pipeline exports the recordings as Playwright, Selenium, Robot Framework or pytest code and runs that in GitHub Actions, GitLab CI, Jenkins or Azure DevOps. The monitor side is in the platform: API collections on a schedule with the four alert channels above, and folders of recordings on a schedule. The UI/UX validation run also writes a gate verdict, pass, tracked or fail, onto the run, which a CI job can poll and act on; that is data for your pipeline to read, not a block the platform applies.

## For the manager reading this

Count the production failures of the last quarter that customers reported before you did. Each one is a question your pre-deploy suite was never asked. The monitor costs a schedule and an alert channel; the recordings and collections you already have are the definition. A 14-day trial on your real journeys shows the first production-only failure the monitor catches, if there is one to catch.

## What to do next

1. Count the production failures of the last quarter that customers reported before you did; each one is a question the pre-deploy suite was never asked.
2. Record the money journeys once against a test account; the folder is the pre-deploy gate, run on demand or on a schedule.
3. Run the API collection behind the same journeys on a schedule as a production monitor, with assertions on status, body and timing and an alert channel ([CI/CD integration](https://dokimore.com/documentation/ci-cd-integration) says what alerting exists today).
4. Include a request that depends on each third party you cannot control, so their bad afternoon is seen by the monitor first.
5. Keep the production monitor read-mostly: sign in, search, read; a monitor that writes needs synthetic data and a cleanup.
6. Treat a failing monitor as an incident and a failing gate as a blocked branch; the same assertion means a different action in the two places.

## Questions people ask

### What is the difference between synthetic monitoring and end-to-end testing?

The environment and the question. End-to-end testing runs a user journey against a build on an environment you control, before the release, to catch regressions in the code. Synthetic monitoring runs a journey against production on a schedule, after the release, to catch failures in configuration, certificates, dependencies and the network that no staging environment contains. The SREcon24 EMEA talk from Elastic argues they should share one definition of the journey.

### Can the same flow be the monitor?

In Dokimore the flow is the shared definition: its recording replays as the regression test before the deploy, and the API collection behind the same flow runs on a schedule as a monitor after it, with alerts to Slack, Microsoft Teams, Opsgenie or a webhook. Journeys that only a browser exercises, such as a single-sign-on redirect, are recorded against production with a test account and scheduled in their own folder.

### Does Dokimore run the pre-deploy gate inside my CI pipeline?

Not from the platform yet: there is no CI trigger, CI token or quality gate. Export the recordings as Playwright, Selenium, Robot Framework or pytest code and run that in your pipeline, or run the folder in Dokimore on demand or on a schedule and read the result from the release checklist.

### What should a production monitor never do?

Create real orders, send real emails or touch real customer data. A monitor that signs in with a test account, searches and reads a page catches most production-only failures without side effects; a monitor that must write should use synthetic data and clean up after itself.

### What was the Cloudflare outage of 18 November 2025?

A configuration file for Cloudflare’s Bot Management product doubled in size after a database permissions change, crossed a 200-feature limit in the software reading it, and the network returned errors from 11:20 UTC until core traffic recovered at 14:30. Cloudflare states it was not an attack. It is the clearest recent case of a failure that no customer’s staging environment could have shown and a scheduled production monitor would have caught.

## How we measured this

- The team, its deploy frequency and its three failures are a scenario chosen so the sequence is concrete, not a measurement of any customer.
- Dokimore’s behaviour is the product on the article date: folders of recordings on demand or on a schedule (scheduled runs use Chromium), the execution grid on Team, API collections as scheduled monitors with Slack, Microsoft Teams, Opsgenie and webhook alerts, code export in four frameworks, a gate verdict written on UI/UX validation runs, and no CI trigger from the platform.

## Sources

1. [USENIX SREcon24 EMEA, Carly Richmond (Elastic), "Synthetic Monitoring and E2E Testing: 2 Sides of the Same Coin" (30 October 2024)](https://www.usenix.org/conference/srecon24emea/presentation/richmond) (read 2026-10-28)
2. [Cloudflare, "Cloudflare outage on November 18, 2025": 11:20 UTC start, 14:30 core recovery, feature file doubled past a 200-feature limit, not an attack](https://blog.cloudflare.com/18-november-2025-outage/) (read 2026-10-28)

Source: https://dokimore.com/blog/release-verification-one-flow-before-and-after-deploy
