Lint makes me pick exactly one, which is fine, but the three names do not tell me much and everyone on the team has a different theory. Right now roughly everything is @regression and @smoke has four scenarios in it that somebody picked at random in the first week.
Is there a definition that survives contact with a real suite, or is this just taste?
The definitions that hold up are about when the suite runs and what happens when it goes red, not about how important the feature is.
@smoke is a budget. It runs on every pull request, on one browser, and it has to stay under a few minutes. If adding a scenario to it would push it over, something else has to leave. Being in smoke means "this may block a merge".@regression is the truth. Everything with product value, run nightly on every browser. This is the default; a new scenario is regression unless there is a reason.@sanity is targeted. A small set you point at something after a hotfix or a data migration, when you want an answer in ninety seconds and you do not want the nightly.The budget framing is what makes the smoke set stable. Without it people add "just this one" until smoke is twenty minutes and gets skipped.
Two practical consequences that made the choice concrete for us.
The suite tag selects the screenshot policy, so the choice has a cost:
screenshots:
policy:
default: on-failure
'@smoke': scenario
'@regression': step
'@sanity': scenario@regression captures a screenshot per step. That is what you want when the nightly fails and nobody was watching; it is not what you want on a pull request.
And the names are configurable. tags.suites is a list, and lint enforces "exactly one of whatever is in it". If your team genuinely thinks in critical/extended you can say so — the lint message then names your suites.
We renamed ours and regretted it. Every doc page, every example and every new joiner arrives expecting smoke/regression/sanity, and you spend the rest of the project translating. If the default three can be made to fit, make them fit.
Have the same question?
Ask it with the command you ran and the output you got, and it will be answered here.
Ask a question