Example threads. These questions and answers were written by the SDODS team to document real problems and how to fix them. The people named are illustrative, not real users. Questions asked on this site are marked new.
sdods run -p demo-shop -e staging --process release-gate✖ failed 318 passed, 0 failed, 0 skipped, 2 flaky (662s)Nothing failed. Zero. And the job is red, so the release is blocked and I am the one holding it.
I can see the two flaky scenarios in the report and they both ended up passing. What is failing the run?
Accepted answer
The gate is. That is what a gate is for.
release-gate in the demo YAML carries failOnFlaky: true and gates: { minPassRate: 100, maxFlaky: 0, perfBudgets: true, a11y: true }. Two flaky scenarios is maxFlaky: 0 violated, so the process fails even though every scenario eventually passed.
That is deliberate, and it is the difference between a report and a decision: a scenario that only passed on a retry has not demonstrated that the release is good, it has demonstrated that the release is good about two thirds of the time. For a nightly you would absorb that. For a release you should not.
Two legitimate ways forward, and one that is not:
maxFlaky: 1 is your standard and change it in the YAML, where it is reviewed.--retries 3 on the command line to make it go away. Flags override the process's fields, which is exactly how a gate gets quietly weakened.To read it yourself next time: summary.json has the totals including flaky, and run.json records which process the run used. Between the two you can tell "scenarios failed" from "the gate was not met" without opening the HTML report.
Exit code is 1 in both cases, which is the one thing here that is arguably too coarse.
Wrap commands and output in triple backticks and they will render as a code block.
Checking whether you’re signed in…
Have a different question? Ask it.