One scenario fails maybe one run in six. Every time I raise it I am told it passed locally, and every time I run it locally it passes, so I am the person crying wolf about the checkout test.
I want a number I can put in a ticket. Is there a way to run one scenario many times?
Accepted answer
Yes, and this is the exact command for it:
sdods run -p demo-shop -e staging --scenario "Checkout with a saved card" --repeat-each 20 --fail-on-flaky--scenario matches scenarios whose title contains that text, --repeat-each runs each of them twenty times, and --fail-on-flaky makes the run exit 1 if any of them passed only on a retry. Twenty green runs is an argument; three failures out of twenty is a ticket.
Once you have the number, classify before you fix: heal events in the run point at locator drift, API snapshots with 5xx or timeouts point at the environment, and the before/after screenshots usually show a timing race. The fix differs for all three, and only the last one is really "the test's fault".
If twenty repeats come back clean, try reproducing the conditions rather than the scenario: raise -w so it runs alongside its neighbours. A good share of the flakiness we chase only exists in parallel — shared data, a pool user two scenarios both wanted, an application that does not like two sessions.
Have a different question? Ask it.