Someone renamed a tag and our pipeline step is:
sdods run -p demo-shop -e staging -t @smoke-uiExit code 0, green check, everyone happy. I only looked because a bug walked into production that we have a scenario for. Scrolling back through the log:
⚠ No scenarios matched the selection (tags @smoke-ui · layers ui,api,hybrid).
✔ passed 0 passed, 0 failed, 0 skipped, 0 flaky (4s)Zero passed is not passed. Why is that a zero exit code?
Accepted answer
Because an empty selection is normal in the shape CI usually has: a shard or a matrix leg with nothing to do has to stay green, or a three-browser matrix fails the moment one browser has no matching scenario. So the run warns and exits 0.
For a command a person scoped by hand that reading is wrong, and today the warning is the only signal you get. Two habits that catch it:
--list before you commit a new selection, so you see the tests rather than trusting the tag.sdods features list -p demo-shop for the real inventory — module, layer and tags per feature. That is the table to look at when an expression selects nothing.An unknown tag on the command line is not an error and lint will not save you: lint checks the tags in your features, not the ones you type.
We got bitten by exactly this and now the pipeline asserts a floor. After the run:
test "$(jq .totals.total .sdods/runs/$RUN_ID/summary.json)" -gt 0Crude, but it has caught two more renames since. The summary is written before ingest, so it is there whether or not you have a database.
Late to this, but the root cause is worth naming: nobody should be renaming a suite tag. There are three of them, they are the interface to the suite, and the whole CI estate is written against those names. If you genuinely need a fourth, add it under tags.suites in the project YAML rather than inventing @smoke-ui in one feature file.
Have the same question?
Ask it with the command you ran and the output you got, and it will be answered here.
Ask a question