Scenarios for a feature flag that is on in staging and off in production

Asked 30 Jun 20263,420 views2 answers

16votes

Our new checkout is behind a build-time flag. On staging it is on, on the production build it is off, and the flag is baked into the bundle at build time so there is nothing to query at run time.

Eleven scenarios cover it. On staging they pass. On the post-deploy sanity run against production they all fail, correctly — the feature is not there.

I tried checking for the feature inside a step and skipping if it is absent. That felt wrong the moment I wrote it. What is the intended way to say "these scenarios belong to one environment"?

ci config regression

asked by Zeynep Arslan64 · 30 Jun 2026

2 answers

22votes✓
Accepted answer
Your instinct about the step is right — delete that. A scenario that decides at run time whether to assert is a scenario that can never fail, and it will go green for the wrong reason long before anyone notices.
There is no flag-aware selection today. Nothing in SDODS talks to a flag service, and because your flag is baked at build time there is not even a run-time fact to read. That is the honest state of it.
The closest supported thing is @env:<name>, which restricts a scenario to one environment and is validated against envs.available:
gherkin
```
  @ui @regression @env:staging
  Scenario: The new checkout summarises the order before payment
```
The cost is that it is hand-maintained. When the rollout flips, somebody edits eleven feature files, and nothing reminds them. It also collapses if the same flag is on in two environments and off in a third — you end up with a tag expression instead of a tag.
For that case, tag the group and exclude it explicitly per run:
bash
```
sdods run -p demo-shop -e prod -t "@sanity and not @newcheckout"
```
Declare newcheckout under tags.extra so lint accepts it rather than warning.
answered by Reeta Nandalmaintainer41,280 · 1 Jul 2026
14votes
Put the exclusion in a process rather than in the command, whichever of the two mechanisms you choose:
yaml
```
processes:
  - name: prod-sanity
    trigger: manual
    env: prod
    tags: '@sanity and not @newcheckout'
```
Two reasons. It is auditable — when someone asks why eleven scenarios did not run against production, the answer is a line in a reviewed file rather than a flag in a workflow. And when the flag ships everywhere, it is one deletion in one place instead of a search across your CI files.
We keep a comment above each such exclusion naming the flag and the person who owns it. Ours have all been removed within a quarter, which is the point.
answered by Rasha Halabi572 · 2 Jul 2026

Have the same question?

Ask it with the command you ran and the output you got, and it will be answered here.

Ask a question

Scenarios for a feature flag that is on in staging and off in production

2 answers

Related