Questions tagged smoke
The fast suite that gates a pull request.
13 questions
Example threads. These questions and answers were written by the SDODS team to document real problems and how to fix them. The people named are illustrative, not real users. Questions asked on this site are marked new.
Does -t smoke,sanity mean both tags or either tag?
Reading a colleague's script: No at-signs, a comma, no quotes. It runs more scenarios than -t @smoke alone, so I assume it is an or — but I want to be sure it is not something…
asked by Zeynep ArslanA renamed tag left our smoke job passing with zero tests for three weeks
Three weeks. The only trace, in a log nobody opens: I understand from an older thread here that an empty selection exits 0 on purpose so a matrix leg with nothing to do stays…
asked by Tanvi BhattRunning the smoke suite on a plane — what do I need so nothing touches the network?
Long flight, and separately we are being asked to prove the suite runs on an air-gapped runner. Same problem twice, so I would like to get it right once. Everything is tagged…
asked by Rasha HalabiOur @smoke check takes fourteen minutes and people have stopped waiting for it
120 scenarios tagged @smoke, one browser, fourteen minutes on the pull request. The team merges and then reads the result, which means it is not a gate any more. I do not want to…
asked by Tanvi BhattA typo in -t made CI green for three weeks — why does matching nothing exit 0?
Someone renamed our pull-request step and dropped a letter: Three weeks of green pull requests. Nothing ran. The output even says so if you scroll up: A green tick under a…
asked by Bruno TeixeiraMy page object needs a signed-in session before goto() — where does the login belong?
Every one of our scenarios opens with the same three steps: go to login, fill two fields, submit. Then the real scenario starts. It is slow — about four seconds a scenario across…
asked by Helle BorgFirst UI run: five scenarios fail on the sign-in page, but the API layer was green
Working through the workshop on my own checkout of the sample app. The API layer passed straight away. Then the browser: Five failures, and as far as I can tell they are all the…
asked by Yusuf DemirHow do you actually decide between @smoke, @regression and @sanity?
Lint makes me pick exactly one, which is fine, but the three names do not tell me much and everyone on the team has a different theory. Right now roughly everything is…
asked by Sunita KaleWhat should a pull request run, and what should wait for the nightly?
We have about 400 scenarios and currently every pull request runs all of them on three browsers. It takes 35 minutes and people merge without waiting, which makes the whole thing…
asked by Elsa NymanLint says "found @smoke @regression" — why can a scenario not be in both suites?
I tagged our happy-path checkout scenario with both suite tags because I want it in the pull-request smoke run and in the nightly regression run. Lint disagrees: Surely wanting a…
asked by Annika SolheimCI has been green for a week and I do not think the suite ran at all
Someone renamed a tag and our pipeline step is: Exit code 0, green check, everyone happy. I only looked because a bug walked into production that we have a scenario for.…
asked by Karine MoreauDo -l, -b and -t stack, or does the last one win?
I have been typing this without really knowing what it means: Three filters. Do they narrow each other down, or is one of them authoritative? Specifically: I have a scenario…
asked by Avery Hollissdods run -l api finished in under two seconds — did it actually run anything?
I pointed the CLI at staging and asked for the API layer only: It printed nine passing scenarios and was finished before I had read the command back. No browser window, no window…
asked by Dinuka Perera