← All questions

Every scenario passed and the release gate still failed the job

sdods run -p demo-shop -e staging --process release-gate

✖ failed  318 passed, 0 failed, 0 skipped, 2 flaky  (662s)

2 answers

• 38votes✓

  Accepted answer

  The gate is. That is what a gate is for.

  release-gate in the demo YAML carries failOnFlaky: true and gates: { minPassRate: 100, maxFlaky: 0, perfBudgets: true, a11y: true }. Two flaky scenarios is maxFlaky: 0 violated, so the process fails even though every scenario eventually passed.

  That is deliberate, and it is the difference between a report and a decision: a scenario that only passed on a retry has not demonstrated that the release is good, it has demonstrated that the release is good about two thirds of the time. For a nightly you would absorb that. For a release you should not.

  Two legitimate ways forward, and one that is not:

  • Fix the two scenarios, which is the point of the gate telling you.
  • Decide as a team that maxFlaky: 1 is your standard and change it in the YAML, where it is reviewed.
  • Do not pass --retries 3 on the command line to make it go away. Flags override the process's fields, which is exactly how a gate gets quietly weakened.

  RHanswered by Rasha Halabi572 · 16 Sept 2025
• 13votes

  To read it yourself next time: summary.json has the totals including flaky, and run.json records which process the run used. Between the two you can tell "scenarios failed" from "the gate was not met" without opening the HTML report.

  Exit code is 1 in both cases, which is the one thing here that is arguably too coarse.

  RNanswered by Reeta Nandalmaintainer41,280 · 17 Sept 2025