Intermittent feedback damages decisions
A flaky test produces different outcomes without a relevant product change. The inconsistency can come from the product, test code, data, environment, or infrastructure.
Repeated false alarms teach people to ignore red pipelines. Blind retries can then hide both test defects and intermittent product failures.
Treat the event as a defect in a production feedback system. Record frequency, conditions, owner, impact, and current containment.
Capture evidence before rerunning
Preserve the trace, screenshot, video, network log, console output, timing, worker, browser, commit, and test-data identity when available.
Compare failing and passing runs. Change one variable at a time, such as worker count, locale, clock, dependency response, or data setup.
Do not diagnose only from the final assertion. The root cause may start earlier through shared state, an unawaited operation, or a stale locator.
Contain without hiding
Quarantine can protect the main signal when the test has an owner, visible issue, review date, and separate execution path.
A retry can collect evidence about frequency, but the first failure must remain visible in reports. Passing after retry is not the same as stable feedback.
Delete a check when its value is lower than repair cost and no important risk becomes uncovered. Replace it at another level when that gives a better signal.
Prevent common instability
Control test data and isolate actors. Wait for observable application conditions instead of fixed delays. Keep tests independent of execution order.
Mock only boundaries that the check does not need to prove. An unrealistic test double can make stability look better while weakening the product claim.
Review the flake rate by suite and cause. Use recurring failures to improve product testability, environment health, and automation design.