Test reliability: flaky tests, self-healing and debugging
A test suite is only as useful as its red builds are believable. The studies that measured flaky tests found the same first cause every time, a test that does not wait properly for something asynchronous, and the same expensive habit: retrying until green and moving on. Reliability work is finding the cause, fixing it at the root, and keeping a flake rate per test so the next one is caught early.
Two other things erode trust in a suite: a locator that breaks because the page changed, which is maintenance rather than flakiness, and a failure nobody can read because the run left no evidence. The pages below cover all three: the flaky-test guide with its numbers, the self-healing guide that shows every rescue, and the debugging guide that says what to capture on every run.
What people search for
The literal queries this hub is written to answer; each leads to one of the pages below.
- flaky tests
- how to fix flaky tests
- flaky tests playwright
- self healing test automation
- test failure analysis
- how to debug playwright tests
- element not found error
Learn: what the studies measured
Numbers from the papers and the engineering posts, quoted with their dates.
How to fix flaky tests
45% of fixes were async waits; what each cause needs; retries and quarantine done honestly.
Flaky test
The definition the tools encode.
Element drift
Why a locator stops matching when nothing is broken.
Self-healing with a person in the loop
Why a rescue is shown rather than applied silently.
Do it: find the cause
Executed guides.
How to debug a failed Playwright test
The timeout message decoded, the trace, console and network signals, from a run we recorded.
XPath vs CSS selectors
Which selectors survived a row inserted above the target, in three engines.
Self-healing and heal review
How a replay rescues a broken locator and where it shows you.
Smart test selection
Run what a change touches, with a written rationale per pick.
Prove it: in the product
Where reliability is measured on your own runs, not asserted.
The test recorder
Several locators stored per element; each rescue shown with its strategy and confidence.
Test evidence
Screenshots, network calls and the assertion result on every step.
XPath and CSS selector tester
Try the locator before you trust it.
Questions people ask
What is the most common cause of a flaky test?
A test that uses the result of something asynchronous before it is ready: 45% of the fixes in the first large study (Luo et al., 2014) and 45.1% of 235 flaky UI tests in a 2021 study. Waiting for a condition instead of a fixed time is the fix in most of those cases; the flaky-test guide covers the others.
Is a test that fails because the page changed a flaky test?
No. A flaky test passes and fails on the same code; a test whose locator stopped matching fails every time until the locator is updated. That is maintenance, and it is what self-healing addresses: a fallback locator stored at record time rescues the step and the run shows the rescue so a person can check it.
Does retrying a failed test fix it?
A retry that passes proves the test is flaky and buys time; it does not fix anything. Keep retries visible as a flaky status that can fail the run, measure a flake rate per test, and fix the cause. Software Engineering at Google puts the point where tests begin to lose value at about 1% flakiness.
Try it on your own application
The Free plan asks for no card and runs for 14 days after the trial: record a flow, replay it, and read the evidence each step produced.