Example threads. These questions and answers were written by the SDODS team to document real problems and how to fix them. The people named are illustrative, not real users. Questions asked on this site are marked new.
Followed the flaky triage advice and quarantined the worst offender:
sdods insights quarantine <fingerprint>It shows as quarantined on the dashboard. It also still fails the nightly, and the gate is still red, which is the entire thing I was trying to avoid. Do I need to pass something to run for the quarantine to be picked up?
There is nothing to pass. sdods run does not read the quarantine list — the flag is recorded, insights and the dashboard show it, and the runner never consults it. So a quarantined scenario runs exactly as before and counts exactly as before.
Open issue. What it needs is for the runner to read the quarantine state and skip or soft-report those scenarios; that has not been written.
Meanwhile the only lever is tags. Keep the quarantine flag for the reporting (it is genuinely useful on the dashboard) and add a @quarantine tag to the scenario as well, then run two selections:
sdods run -p <slug> -t "@regression and not @quarantine" --fail-on-flaky
sdods run -p <slug> -t "@quarantine" --no-ingestThe first is the gate. The second keeps the quarantined scenarios executing so you still get the flake signal and notice when one starts passing again.
I will be honest that this is not a good workaround. You now have two sources of truth for which scenarios are quarantined, and they drift.
On the drift: put the exclusion in a named process rather than in each workflow, so the gate and anyone running locally use the same expression. Ours drifted within a fortnight when it lived in two workflow files — someone quarantined a scenario, added the tag, and only the nightly had the exclusion.
It does not fix the two-sources problem, it just stops it multiplying.
Wrap commands and output in triple backticks and they will render as a code block.
Checking whether you’re signed in…
Have a different question? Ask it.