Followed the flaky triage advice and quarantined the worst offender:
sdods insights quarantine <fingerprint>It shows as quarantined on the dashboard. It also still fails the nightly, and the gate is still red, which is the entire thing I was trying to avoid. Do I need to pass something to run for the quarantine to be picked up?
There is nothing to pass. sdods run does not read the quarantine list — the flag is recorded, insights and the dashboard show it, and the runner never consults it. So a quarantined scenario runs exactly as before and counts exactly as before.
Open issue. What it needs is for the runner to read the quarantine state and skip or soft-report those scenarios; that has not been written.
Meanwhile the only lever is tags. Keep the quarantine flag for the reporting (it is genuinely useful on the dashboard) and add a @quarantine tag to the scenario as well, then run two selections:
sdods run -p <slug> -t "@regression and not @quarantine" --fail-on-flaky
sdods run -p <slug> -t "@quarantine" --no-ingestThe first is the gate. The second keeps the quarantined scenarios executing so you still get the flake signal and notice when one starts passing again.
I will be honest that this is not a good workaround. You now have two sources of truth for which scenarios are quarantined, and they drift.
On the drift: put the exclusion in a named process rather than in each workflow, so the gate and anyone running locally use the same expression. Ours drifted within a fortnight when it lived in two workflow files — someone quarantined a scenario, added the tag, and only the nightly had the exclusion.
It does not fix the two-sources problem, it just stops it multiplying.
Have the same question?
Ask it with the command you ran and the output you got, and it will be answered here.
Ask a question