Skip to main content
Free plan, no cardStart free
Topic · updated 2026-11-04

Test reliability: flaky tests, self-healing and debugging

A test suite is only as useful as its red builds are believable. The studies that measured flaky tests found the same first cause every time, a test that does not wait properly for something asynchronous, and the same expensive habit: retrying until green and moving on. Reliability work is finding the cause, fixing it at the root, and keeping a flake rate per test so the next one is caught early.

Two other things erode trust in a suite: a locator that breaks because the page changed, which is maintenance rather than flakiness, and a failure nobody can read because the run left no evidence. The pages below cover all three: the flaky-test guide with its numbers, the self-healing guide that shows every rescue, and the debugging guide that says what to capture on every run.

What people search for

The literal queries this hub is written to answer; each leads to one of the pages below.

  • flaky tests
  • how to fix flaky tests
  • flaky tests playwright
  • self healing test automation
  • test failure analysis
  • how to debug playwright tests
  • element not found error

Learn: what the studies measured

Numbers from the papers and the engineering posts, quoted with their dates.

Do it: find the cause

Executed guides.

Prove it: in the product

Where reliability is measured on your own runs, not asserted.

Questions people ask

What is the most common cause of a flaky test?

A test that uses the result of something asynchronous before it is ready: 45% of the fixes in the first large study (Luo et al., 2014) and 45.1% of 235 flaky UI tests in a 2021 study. Waiting for a condition instead of a fixed time is the fix in most of those cases; the flaky-test guide covers the others.

Is a test that fails because the page changed a flaky test?

No. A flaky test passes and fails on the same code; a test whose locator stopped matching fails every time until the locator is updated. That is maintenance, and it is what self-healing addresses: a fallback locator stored at record time rescues the step and the run shows the rescue so a person can check it.

Does retrying a failed test fix it?

A retry that passes proves the test is flaky and buys time; it does not fix anything. Keep retries visible as a flaky status that can fail the run, measure a flake rate per test, and fix the cause. Software Engineering at Google puts the point where tests begin to lose value at about 1% flakiness.

Try it on your own application

The Free plan asks for no card and runs for 14 days after the trial: record a flow, replay it, and read the evidence each step produced.

Try it against your own app.

Record one flow and get four kinds of evidence back. Judge the results yourself.

  • Free plan, no card
  • 14-day Solo Pro trial
  • Demo replies within one business day
  • Data hosted in India