Genuine question, not a troll. Our nightly had six or seven failures a week, all different, all "it passes when I run it". We set retries to 3 and it has been green for nine days. Two people have told me this is bad and neither could say why beyond "it hides things".
What is it hiding, concretely?
Accepted answer
Concretely: the pass is now conditional on a retry, and nothing in your green check says how conditional.
A retry does not repair a scenario. It converts a failure into a "flaky" result, and flaky is a state your dashboard tracks precisely because it is not the same as passed. What you have bought is that nobody has to look at the seven weekly failures, which were seven weekly pieces of information about your application or your environment.
Two things to put back:
--fail-on-flaky (or failOnFlaky: true) on the gate that actually matters — the release gate. Then the nightly can absorb a retry and the release cannot.After that, rank what is left by flakiness in insights and fix the top two. That is a smaller job than it sounds; flakiness concentrates.
The routine that works, in order: detect (insights ranks the last 30 runs), classify (locator drift, environment or timing), contain (quarantine so the gate stays honest while you work — the scenario keeps running and reporting), fix, then verify with --repeat-each 10 --fail-on-flaky and lift the quarantine.
The step people skip is "contain". Without it the pressure to raise retries comes back the following week.
Our numbers, for calibration: retries 1 in CI, 0 locally, and a scenario that needs more than one is not a flaky test, it is a bug report against either the app or the scenario. We have about 4,000 scenarios and roughly a dozen carry that debt at any time.
Have the same question?
Ask it with the command you ran and the output you got, and it will be answered here.
Ask a question