We quarantine flaky scenarios in the web UI while somebody fixes them. The dashboard shows them as quarantined, so I assumed that was the end of it.
The release gate disagrees. The quarantined checkout scenario ran last night, failed, and the job exited 1. The gate job does not have a database configured — it uploads artefacts and that is all.
Am I holding this wrong, or does the runner not know about quarantine?
Accepted answer
The runner does not know. This is a real gap, not a misconfiguration on your side.
Quarantine is a fact stored in the database. It is honoured where the numbers are computed — the dashboard, insights, anything reading ingested runs — and it means "keeps running and reporting, does not block". sdods run selects scenarios from one input only: the tag expression. So on a gate job with no database, nothing in the pipeline has ever heard the word quarantine.
The workaround we use, and it is a workaround:
processes:
- name: release-gate
trigger: release
tags: '(@smoke or @regression) and not @quarantine'
failOnFlaky: true
gates: { minPassRate: 100, maxFlaky: 0 }Then tag the scenario @quarantine in the feature file when you quarantine it in the UI, and declare quarantine under tags.extra so lint does not warn about an unknown tag.
The cost is that it is two actions instead of one, and you have to repeat and not @quarantine in every process, schedule and hand-written command that must stay green. We keep it out of the nightly deliberately — the nightly should still run them, because that is how you find out they are fixed.
Confirming the shape of it: selection happens before anything database-backed is consulted, and there is no plan yet for the runner to query the database before choosing scenarios. It would make a run depend on a database being reachable, which is a trade nobody has wanted to make for a gate.
One correction to the docs while you are here: they mention an insights quarantine <fingerprint> command. The insights CLI is a placeholder today — computing and showing work, the toggle does not. Use the web UI. The tag mirror in the answer above is the part that actually changes what runs.
Have the same question?
Ask it with the command you ran and the output you got, and it will be answered here.
Ask a question