# How to Fix Flaky Tests: Find the Cause Before You Retry

Dokimore Support Team · October 20, 2026, updated 2026-11-01 · 13 min read

Format: How-to guide · Written for: SDETs and automation engineers, QA leads and test managers, Developers who own their tests · Where it applies: every market

> A retry that passes proves the test is flaky. It does not fix it. Here is what the studies measured, how to find the cause, and what Playwright, Cypress and Selenium ship for the job.

## In short

- A flaky test passes and fails on the same code. Prove it by rerunning on the same commit, then find the cause: in both studies that classified causes, a test that does not wait properly for something asynchronous comes first, at 45% of fixes in the first large study and 45% again in a later study of UI tests.
- Fix the cause, not the symptom: wait for a condition rather than a time, give each test its own state, control the clock, stub services you do not control, give the test the machine it was written on, and when the race is in the product, fix the product; 24% of fixes in the 2014 study changed the code under test, and 94% of those fixed a real bug.
- Retries and quarantine buy time and should be visible: every quarantined test gets an owner, a ticket and an end date, keeps running outside the gate, and comes back when it stays healthy. GitHub cut red builds caused by flaky tests from 1 in 11 commits to 1 in 200 once it managed them this way.

A flaky test passes and fails on the same code. Nothing changed between the green run and the red one, so the failure says nothing about your change, and every time someone clicks retry and moves on, the team learns that a red build can be ignored. That lesson is the expensive part.

The short answer to how to fix flaky tests: prove the test is flaky by running it again on the same commit, work out which cause you are looking at (in both studies below that classified causes, Luo 2014 and Romano 2021, a test that does not wait properly for something asynchronous comes first), fix that cause instead of adding a pause, and keep a flake rate per test so the next one is caught early. Retries and quarantine buy time. Neither is the fix.

## What counts as a flaky test

Google’s Testing Blog uses the definition most teams share: a flaky result is "a test that exhibits both a passing and a failing result with the same code." The tools encode the same idea. With retries on, Playwright Test reports a test that failed on its first run and passed when retried as "flaky", separately from "passed" and "failed". Cypress Cloud calls a test flaky when it "passes and fails across retries without any code change".

Keep one distinction from the start. A test that fails every time is not flaky; it is broken, and it needs a different response: read the failure. Flaky means the outcome varies while the code does not.

## What causes flaky tests: what the studies measured

- Waiting badly for something asynchronous. In the first large study of flaky tests (Luo, Hariri, Eloussi and Marinov, FSE 2014), 74 of 161 fixes (45% in the paper’s count) were "async wait": the test started an asynchronous action and used its result before it was ready. In 235 flaky UI tests from 62 web and Android projects (Romano et al., ICSE 2021) the share was 45.1%, and 52.0% in the web projects.
- Concurrency and test order. The same 2014 study found concurrency behind 20% of the fixes and dependence on test order behind 12%; with async waits, those three causes covered 77%.
- Time and randomness. Smaller groups in that study relied on the system clock (a test that fails when midnight passes in UTC) or on random values whose full range the test did not expect.
- The machine running the test. Across 52 projects run under 27 resource settings, Silva et al. (IEEE TSE) found 46.5% of flaky tests were resource-affected: whether they failed depended on the CPU and memory available, which is exactly what changes on a busy CI runner.
- Size. Over one week at Google in 2017, 0.5% of small tests, 1.6% of medium tests and 14% of large tests were flaky. Browser tests through WebDriver were flaky more often than average, which the author attributes mostly to their size.
- Shared dependencies. In 10,000 suite runs of 24 Java projects (Parry et al., EASE 2025), 75% of flaky tests failed together in clusters, mostly because of intermittent networking and unstable external dependencies. One fix can clear a whole cluster.
- The product itself. In the 2014 study, 24% of the fixes changed the code under test, and 94% of those fixed a real bug. A flaky test is sometimes the only witness to a race condition in your application, which is the best argument against deleting one.

## How to find flaky tests before they spread

- Run it again on the same commit. A failure that passes on a rerun with no code change is the working definition. GitHub’s system marked a failure flaky when another build of the same code disagreed or when the test passed on retry, and by retrying under varied conditions it identified 90% of flaky failures automatically (2020).
- Repeat new tests before they merge. Luo et al. found 78% of flaky tests were flaky from the day they were written, so the cheapest place to catch one is the pull request that adds it: run the new test many times in a row.
- Measure flakiness as a rate, not a label. Meta scores every test from its CI history, on the reasoning that "the right question to ask is not whether a particular test is flaky, but how flaky it is", and opens a ticket for the owner when the score trends up (2020).
- Show each team its own numbers. Spotify reports that publishing a table of each team’s fast, slow and flaky tests, and doing nothing else, cut its test flakiness from 6% to 4% in two months (2019).

## How to fix flaky tests, cause by cause

- An async wait: wait for the condition, never for a time. In the 2014 study, 57% of the async-wait tests were fixed with a call that waits for a condition, which the authors call "the most efficient and effective way" to fix them; adding or lengthening a sleep only lowered the chance of failure. A 2025 study of 49 async-wait web tests (Pei et al.) found developers adjusted a wait time in about 63% of fixes even when the cause lay elsewhere, which is how the same test comes back next month.
- Shared state and test order: give each test its own world. 74% of the order-dependent tests in the 2014 study were fixed by cleaning shared state between runs. Better still, create the state each test needs inside the test, and never rely on another test having run first.
- Time: control the clock. Martin Fowler’s advice is to "always wrap the system clock, so it can be easily substituted for testing". In a browser test, freeze or advance the page’s clock instead of waiting for it.
- Services you do not control: stub them. Playwright’s best-practices page says to test only what you control and to return the response you need from third-party calls through its network API, instead of depending on a service outside the test.
- Resources: give the test the machine it was written on. Playwright’s CI guide recommends one worker in CI so each test gets the full machine, with parallel runs only on powerful self-hosted runners or by sharding across machines.
- The application: when the race is in the product, fix the product. If the change belongs in the code under test, file it as a bug, not as test maintenance.

## Flaky tests in Playwright: what is built in

- Retries with a flaky status. Set retries in the config and Playwright reports each test as passed, flaky or failed. After a failure it discards the worker process with its browser and retries in a new one.
- Failing the run on flakiness. The --fail-on-flaky-tests option (1.45; failOnFlakyTests in the config since 1.52) exits with an error when any test is flagged flaky, so retries can keep a suite moving without hiding the flake.
- A trace of the failure. With trace set to on-first-retry, Playwright records a trace when a failed test is retried, which the Trace Viewer opens step by step with the page and the network at the moment it went wrong.
- Repeat and re-run. --repeat-each runs every test N times, which is how you prove a fix holds; --last-failed (1.44) re-runs only the failures, and --only-changed (1.46) runs only the test files changed since a Git reference.
- Waiting done for you. Before each action Playwright waits until the element is visible, stable, able to receive events and enabled, and assertions such as expect(locator).toBeVisible() retry until they pass or time out (5 seconds by default). The docs mark page.waitForTimeout() as discouraged because "tests that wait for time are inherently flaky".
- Isolation and control. Each test gets its own browser context; role locators (getByRole, since 1.27) follow what users see rather than the markup; the Clock API (1.45) controls time; page.route stubs network calls. Serial mode retries a whole group from the start, so keep it for tests that genuinely depend on each other.

## Cypress and Selenium

Cypress does not retry failed tests by default. Its retries setting takes separate counts for cypress run and cypress open, and beforeEach and afterEach hooks run again on each attempt. Its best-practices page lists cy.wait() with a fixed number as an anti-pattern and recommends waiting on route aliases or assertions. Flake detection, the flaky label and a flake rate over recent runs are Cypress Cloud features that work on runs recorded from CI with retries on.

Selenium’s documentation calls the race between the browser reaching the right state and the test code running first "one of the primary causes of flaky tests", and its answer is an explicit wait for a condition. It also warns: "Do not mix implicit and explicit waits"; a 10-second implicit wait with a 15-second explicit one can time out after 20 seconds.

## Retries and quarantine, without losing the signal

Retries are a trade, not a fix. Software Engineering at Google describes an automatic rerun as "trading CPU cycles for engineering time", sensible at low levels of flakiness, and adds that "rerunning a test is only delaying the need to address the root cause of flakiness." The same book says tests begin to lose value as flakiness approaches 1%.

Quarantine takes a flaky test out of the gate without taking it out of sight. Google quarantines a test automatically when its flakiness is too high and files a bug, while warning that quarantine "could easily mask a real race condition or some other bug in the code being tested." Slack’s automatic suppression, with a Jira ticket for each suppressed test, cut test-job failures from 57% to under 5% (2022). Atlassian keeps running quarantined tests outside the main pipeline, gives the owning team a ticket with a due date, and returns a test once it stays healthy for a set period (2025).

- Every quarantined test has an owner, a ticket and an end date.
- Quarantined tests keep running and reporting, outside the gate.
- A test that fails on every run is broken. It does not belong in quarantine; it needs reading.
- Delete a flaky test only when you can show it protects nothing. The 2014 study’s 94% says it often does.

## What flakiness costs

At Google in 2016, about 1.5% of all test runs reported a flaky result, almost 16% of tests had some flakiness, and about 84% of the pass-to-fail transitions observed involved a flaky test. In GitHub’s monolith in 2020, 1 in 11 commits had a red build caused by a flaky test; after GitHub started managing flaky tests, 1 in 200 did.

Atlassian reports flaky tests behind as much as 21% of master build failures in its Jira Frontend repository, and about 15% of Jira backend failures, with reruns that waste over 150,000 hours of developer time a year (2025). An industrial case study at ICST 2024 measured at least 2.5% of productive developer time going to flaky tests, and put an automatic rerun at 0.02 cents against $5.67 for a manual investigation in that project.

## How Dokimore handles flaky tests

Recorded tests replay on Playwright, so the waiting described above (actions that wait for the element, assertions that retry) is the engine under every step rather than a setting someone has to remember.

Each test folder has a quarantine panel. It reads the last 50 Chromium runs of every file and gives no verdict until a file has run 5 times. A file with at least 2 passes and 2 failures in that window is flaky; one that keeps failing with no pass is broken; the rest are healthy. When a flaky file fails at least 20% of the time, the panel recommends quarantine. A person applies it: quarantined files are skipped by folder runs and reruns, and stay listed in the panel until someone takes them out.

A changed page is a different problem. When a locator stops matching because the page changed, a fallback locator may rescue the step, and the run detail shows the original and the replacement, so a rescued locator is never mistaken for a flaky test or applied silently. Every step keeps its screenshot and network calls, which is usually enough to tell a timing failure from a real one.

## What to do next

1. Rerun the failing test on the same commit; a pass with no change is the working definition of flaky, and a test that fails every time is broken, not flaky.
2. Repeat every new test many times in the pull request that adds it; 78% of flaky tests were flaky from the day they were written.
3. Fix the cause, not the symptom: wait for a condition rather than a time, give each test its own state, control the clock, stub the services you do not control.
4. Give the test the machine it was written on: one worker in CI, parallel runs only on runners that can take them.
5. Quarantine with an owner, a ticket and an end date, keep the test running outside the gate, and bring it back when it stays healthy.
6. Keep a flake rate per test from run history and open a ticket when it trends up; [the practice that outlasts its champion](https://dokimore.com/blog/automation-practice-that-lasts) says who should own it.

## Questions people ask

### What is a flaky test?

A test that passes and fails on the same code, with nothing changed between the runs. Google’s Testing Blog defines it that way, and Playwright and Cypress encode the same idea: a test that failed first and passed on retry is reported as flaky, separately from passed and failed. A test that fails every time is broken, not flaky.

### What is the most common cause of flaky tests?

Waiting badly for something asynchronous: the test uses a result before it is ready. It was 45% of the 161 fixes in the first large study (Luo et al., 2014) and 45.1% of 235 flaky UI tests in a 2021 study, 52% in the web projects. Concurrency and test-order dependence come next; together the three covered 77% of fixes in the 2014 study.

### Should I just retry flaky tests?

A retry proves the test is flaky and buys time; it does not fix it. Software Engineering at Google calls automatic reruns "trading CPU cycles for engineering time", sensible at low flakiness, and warns that a rerun only delays the root cause. Keep retries visible, with a flaky status that can fail the run, and fix the cause.

### How does Playwright help with flaky tests?

It waits for an element to be visible, stable and enabled before each action, retries assertions until they pass, reports a retried-then-passed test as flaky, can fail the run on any flaky test, records a trace on the first retry, repeats a test N times to prove a fix, and isolates each test in its own browser context. Fixed-time waits are discouraged in its documentation because they are inherently flaky.

## How we measured this

- Every figure is quoted from the source named next to it, with its year and what was measured. We did not repeat these studies; the papers and engineering posts were read on 2026-10-20, and the exact quotes are kept in our research file. Corrected 2026-10-28: the Pei et al. and ICST 2024 figures were cited in the text without entries in this list; both are listed now.
- Playwright, Cypress and Selenium behaviour comes from their official documentation, read on 2026-10-20. A Playwright version is given only where its release notes say when a feature arrived.
- Dokimore’s behaviour is the product code on the article date: the quarantine panel reads the last 50 Chromium runs of each test file, gives no verdict before 5 runs, calls a file flaky once it has at least 2 passes and 2 failures in that window, and recommends quarantine at a 20% failure rate. A person applies quarantine; the platform never does it alone.

## Sources

1. [Luo, Hariri, Eloussi, Marinov, "An Empirical Analysis of Flaky Tests", FSE 2014: 161 fixes classified](https://mir.cs.illinois.edu/marinov/publications/LuoETAL14FlakyTestsAnalysis.pdf) (read 2026-10-20)
2. [Romano et al., "An Empirical Analysis of UI-based Flaky Tests", ICSE 2021](https://arxiv.org/pdf/2103.02669) (read 2026-10-20)
3. [Silva et al., resource-affected flaky tests, IEEE Transactions on Software Engineering](https://arxiv.org/abs/2310.12132) (read 2026-10-20)
4. [Parry, Kapfhammer, Hilton, McMinn, co-occurring flaky test failures, EASE 2025](https://arxiv.org/abs/2504.16777) (read 2026-10-20)
5. [Google Testing Blog, "Flaky Tests at Google and How We Mitigate Them" (2016)](https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html) (read 2026-10-20)
6. [Google Testing Blog, "Where do our flaky tests come from?" (2017)](https://testing.googleblog.com/2017/04/where-do-our-flaky-tests-come-from.html) (read 2026-10-20)
7. [Software Engineering at Google, chapter 11: reruns trade CPU for engineering time; value lost near 1% flakiness](https://abseil.io/resources/swe-book/html/ch11.html) (read 2026-10-20)
8. [GitHub Engineering, "Reducing flaky builds by 18x" (2020)](https://github.blog/2020-12-16-reducing-flaky-builds-by-18x/) (read 2026-10-20)
9. [Meta Engineering, "Probabilistic flakiness" (2020)](https://engineering.fb.com/2020/12/10/developer-tools/probabilistic-flakiness/) (read 2026-10-20)
10. [Spotify Engineering, test flakiness methods (2019)](https://engineering.atspotify.com/2019/11/test-flakiness-methods-for-identifying-and-dealing-with-flaky-tests) (read 2026-10-20)
11. [Slack Engineering, handling flaky tests at scale (2022)](https://slack.engineering/handling-flaky-tests-at-scale-auto-detection-suppression/) (read 2026-10-20)
12. [Atlassian Engineering, taming test flakiness (2025)](https://www.atlassian.com/blog/atlassian-engineering/taming-test-flakiness-how-we-built-a-scalable-tool-to-detect-and-manage-flaky-tests) (read 2026-10-20)
13. [Pei, Sohn, Habchi, Papadakis, "Non-Flaky and Nearly-Optimal Time-based Treatment of Asynchronous Wait Web Tests" (FSE 2025, journal-first): 49 reproducible async-wait flaky web tests, wait time adjusted in about 63% of fixes](https://conf.researchr.org/details/fse-2025/fse-2025-journal-first/32/Non-Flaky-and-Nearly-Optimal-Time-based-Treatment-of-Asynchronous-Wait-Web-Tests) (read 2026-10-20)
14. [Leinen, Elsner, Pretschner, Stahlbauer, Sailer, Juergens, "Cost of Flaky Tests in CI: An Industrial Case Study" (ICST 2024): at least 2.5% of productive developer time, a rerun at 0.02 cents against $5.67 for a manual investigation](https://conf.researchr.org/details/icst-2024/icst-2024-industry/1/Cost-of-Flaky-Tests-in-CI-An-Industrial-Case-Study) (read 2026-10-20)
15. [Playwright documentation: retries, assertions, actionability, best practices and CI](https://playwright.dev/docs/test-retries) (read 2026-10-20)
16. [Playwright release notes: when each feature arrived (1.27 role locators, 1.44 --last-failed, 1.45 the Clock API and --fail-on-flaky-tests, 1.46 --only-changed, 1.52 failOnFlakyTests)](https://playwright.dev/docs/release-notes) (read 2026-10-20)
17. [Cypress documentation: test retries and Cypress Cloud flaky test management](https://docs.cypress.io/app/guides/test-retries) (read 2026-10-20)
18. [Selenium documentation, Waiting strategies](https://www.selenium.dev/documentation/webdriver/waits/) (read 2026-10-20)
19. [Martin Fowler, "Eradicating Non-Determinism in Tests" (2011)](https://martinfowler.com/articles/nonDeterminism.html) (read 2026-10-20)

Source: https://dokimore.com/blog/how-to-fix-flaky-tests
