The tags table says @env:<name> "restricts a scenario to one environment", and sdods lint is happy with the tag. But:
sdods run -p portal -e staging -t @regressionran all 41 of our @env:local scenarios against staging. Some of them create data. Nothing warned, nothing skipped.
Am I holding it wrong, or does the tag do nothing?
Accepted answer
It does nothing at run time. Lint validates that the name is one of envs.available and that is the end of it — nothing consults the tag while selecting or running scenarios. Compare @skip:<browser>, which really does skip; @env: looks like its sibling and is not.
Open issue. Until it is enforced, you have to do the exclusion yourself, in the tag expression:
sdods run -p portal -e staging -t "@regression and not @env:local and not @env:prod-preview"The important part is not typing that once. It is making it impossible to forget: put the expression in a named process in the project yaml and run the process, so the gate, the nightly and the person debugging on a laptop all get the same selection. An expression that lives in one person's shell history is the same amount of protection as no expression at all.
The part of your message I would not move past: "some of them create data".
If you are relying on @env: to keep scenarios off a production-like environment, you are relying on nothing. Add something structural underneath it:
users: {}in that environment's yaml. With no pool, any scenario reaching I use a leased user with role "..." throws USER_POOL_EXHAUSTED instead of signing in. It fails closed, which is the right direction.
Two caveats and both matter. It only protects scenarios that lease a user — an anonymous POST, a public form submission, a webhook call is not stopped by an empty pool at all. And it is one well-meaning edit away from being gone: somebody adds a single pool row for a legitimate reason and every @user: scenario is armed against that environment silently. Put a comment on the empty map saying why it is empty, and if you can, a CI assertion that it still is.
Worth counting yours before you decide how much this matters:
grep -rho '@env:[a-z-]*' projects/<slug>/features/ | sort | uniq -cWe assumed we had a handful and had sixty-odd, most of them added by people who reasonably believed the tag was doing something. That number is also a decent argument to take to whoever prioritises the fix.
Have the same question?
Ask it with the command you ran and the output you got, and it will be answered here.
Ask a question