A renamed tag left our smoke job passing with zero tests for three weeks
Example threads. These questions and answers were written by the SDODS team to document real problems and how to fix them. The people named are illustrative, not real users. Questions asked on this site are marked new.
Three weeks. The only trace, in a log nobody opens:
I understand from an older thread here that an empty selection exits 0 on purpose so a matrix leg with nothing to do stays green. Fine. But our job is not a matrix leg, it is the check that gates merges, and it went green for three weeks while testing nothing.
Is there anything better available now than grepping our own logs?
Nothing in the runner will do it for you — that is the honest answer. A selection that matches nothing prints a warning and exits 0:
text
⚠ No scenarios matched the selection (tags: @smoke).
Exit 0, a summary with zero scenarios, and a green tick. Which is what you had for three weeks.
The check has to live in the job. --list resolves the selection and prints what would run, without running it, so a scoped job can assert on it before it starts:
The other half is not writing the expression in the pipeline at all. A process names the recipe once in the project, so a renamed tag is a change to a file that gets reviewed rather than a string in CI that nobody reads.
It is opt-in for exactly the reason you already know — a shard or a matrix leg with nothing to do has to stay green — but a run somebody scoped by hand is the opposite case, and calling that "passed" hides the mistake behind a green badge. Put it on every hand-written command in CI.
It is not in the flag table in the reference page yet. It is in --help.
Runs started from the web UI run dialog already pass it — somebody scoped that run by hand, so matching nothing is a mistake rather than a pass. If you have seen a run from the dialog fail with zero tests and wondered why the same selection from the CLI did not, that is the difference.
Belt and braces on top: our release gate asserts totals.total against a floor read from the previous green run, so a selection that halves is caught too, not only one that empties.
Also: stop renaming suite tags. There are three of them and every pipeline, process and schedule you own is written against those names. If a team genuinely needs a fourth, add it under tags.suites in the project YAML — then lint knows about it and the rename is a config change rather than a silent hole in your gate.