A typo in -t made CI green for three weeks — why does matching nothing exit 0?
Example threads. These questions and answers were written by the SDODS team to document real problems and how to fix them. The people named are illustrative, not real users. Questions asked on this site are marked new.
Someone renamed our pull-request step and dropped a letter:
bash
sdods run -p shop -e staging -l ui -t @smok
Three weeks of green pull requests. Nothing ran. The output even says so if you scroll up:
A green tick under a warning that says nothing ran is a bad combination. Why is that not a failure, and what do we put in the pipeline so it never happens again?
There is no flag for it, which is the uncomfortable part of the answer. An empty selection warns on stderr and exits 0:
text
⚠ No scenarios matched the selection (tags: @smok).
The reason it is a warning rather than an error is that an empty selection is legitimately fine in plenty of runs — a shard that gets nothing, --scenario filtering while you iterate, a module with no api-layer scenarios yet in a matrix that covers every module. Failing all of those by default would be worse than the bug you hit.
So the pipeline has to assert it itself. Two ways that work today:
bash
# 1. Ask what would run, and fail if the answer is nothing.
test -n "$(sdods run -p shop -e staging -l ui -t @smoke --list)" || exit 1
# 2. Or treat the warning as fatal.
sdods run -p shop -e staging -l ui -t @smoke 2>&1 | tee run.log
grep -q 'No scenarios matched' run.log && exit 1
The first is better — it costs a second and it fails before anything runs.
But there is nothing interactive about CI, so turn it on there. It is one flag and it is exactly the failure you wanted three weeks ago.
Also worth understanding why lint did not save you: @smok was never in a feature file. It was on the command line. Lint reads features, so a tag that exists only in a pipeline script is invisible to it — nothing is misspelled in the repository.
That is the whole class of bug. Tag typos inside features are caught by lint; tag typos in an expression are caught by whatever you put around the command, or not at all.
The other half of the fix is not to write the expression on the command line. Name it once in the project and let the pipeline call it:
Worth saying that a gate is not a substitute for the check. A pass-rate gate answers "was everything that ran green"; the --list check answers "did anything run". You want both, because the green tick is claiming both.
For the immediate audit: sdods run ... --list on your current pipeline arguments tells you in seconds whether every step selects a non-empty set. We found two more after this thread.
Coming to this late after doing exactly the same thing with -l instead of -t — -l recorded on a project with no recorded specs. Same silent zero, same green.
One thing that helped afterwards: the warning goes to stderr and the summary to stdout, so a pipeline that only captures stdout never shows the line that would have explained it. Worth checking how your CI collects output before you conclude the warning was not printed.