We have about 400 scenarios and currently every pull request runs all of them on three browsers. It takes 35 minutes and people merge without waiting, which makes the whole thing decorative.
I know the tags exist. What I want is the boring answer: what do teams actually put on a pull request, and what waits?
The taxonomy is designed around exactly this split, so use it as intended:
@smoke on every pull request, one browser, and treat it as a budget rather than a category — a few minutes per browser. If it grows past that, something in it belongs in regression.@regression on merge to main and on the nightly, across the browser matrix. That is where the breadth lives.@sanity after a deploy or a hotfix, narrow and targeted.sdods run -p demo-shop -e staging -t @smoke -b chromium # pull request
sdods run -p demo-shop -e staging -t @regression --project-matrix # nightlyThe thing that keeps it honest is that the pull request check has to be fast enough that people wait for it. A 35-minute check is not a gate, it is a suggestion.
From the side that reads the nightly every morning: put the full matrix in the nightly and let it notify, otherwise nobody looks. And do not put anything in the nightly that you would not act on before lunch — a nightly with 30 known failures is the same as no nightly.
Have the same question?
Ask it with the command you ran and the output you got, and it will be answered here.
Ask a question