Scenarios for a feature flag that is on in staging and off in production

Example threads. These questions and answers were written by the SDODS team to document real problems and how to fix them. The people named are illustrative, not real users. Questions asked on this site are marked new.

Our new checkout is behind a build-time flag. On staging it is on, on the production build it is off, and the flag is baked into the bundle at build time so there is nothing to query at run time.

Eleven scenarios cover it. On staging they pass. On the post-deploy sanity run against production they all fail, correctly — the feature is not there.

I tried checking for the feature inside a step and skipping if it is absent. That felt wrong the moment I wrote it. What is the intended way to say "these scenarios belong to one environment"?

ci config regression

asked by Zeynep Arslan

2 answers

Accepted answer
Your instinct about the step is right — delete that. A scenario that decides at run time whether to assert is a scenario that can never fail, and it will go green for the wrong reason long before anyone notices.
There is no flag-aware selection today. Nothing in SDODS talks to a flag service, and because your flag is baked at build time there is not even a run-time fact to read. That is the honest state of it.
The closest supported thing is @env:<name>, which restricts a scenario to one environment and is validated against envs.available:
gherkin
```
  @ui @regression @env:staging
  Scenario: The new checkout summarises the order before payment
```
The cost is that it is hand-maintained. When the rollout flips, somebody edits eleven feature files, and nothing reminds them. It also collapses if the same flag is on in two environments and off in a third — you end up with a tag expression instead of a tag.
For that case, tag the group and exclude it explicitly per run:
bash
```
sdods run -p demo-shop -e prod -t "@sanity and not @newcheckout"
```
Declare newcheckout under tags.extra so lint accepts it rather than warning.
answered by Reeta NandalSDODS team
Put the exclusion in a process rather than in the command, whichever of the two mechanisms you choose:
yaml
```
processes:
  - name: prod-sanity
    trigger: manual
    env: prod
    tags: '@sanity and not @newcheckout'
```
Two reasons. It is auditable — when someone asks why eleven scenarios did not run against production, the answer is a line in a reviewed file rather than a flag in a workflow. And when the flag ships everywhere, it is one deletion in one place instead of a search across your CI files.
We keep a comment above each such exclusion naming the flag and the person who owns it. Ours have all been removed within a quarter, which is the point.
answered by Rasha Halabi

Know the answer?

Wrap commands and output in triple backticks and they will render as a code block.

Checking whether you’re signed in…

Have a different question? Ask it.

Scenarios for a feature flag that is on in staging and off in production

2 answers

Know the answer?

Related