Questions tagged smoke
The fast suite that gates a pull request.
13 questions
Does -t smoke,sanity mean both tags or either tag?
Reading a colleague's script: No at-signs, a comma, no quotes. It runs more scenarios than -t @smoke alone, so I assume it is an or — but I want to be sure it is not something…
asked by Zeynep Arslan64 ·A renamed tag left our smoke job passing with zero tests for three weeks
Three weeks. The only trace, in a log nobody opens: I understand from an older thread here that an empty selection exits 0 on purpose so a matrix leg with nothing to do stays…
asked by Tanvi Bhatt126 ·Running the smoke suite on a plane — what do I need so nothing touches the network?
Long flight, and separately we are being asked to prove the suite runs on an air-gapped runner. Same problem twice, so I would like to get it right once. Everything is tagged…
asked by Rasha Halabi572 ·Our @smoke check takes fourteen minutes and people have stopped waiting for it
120 scenarios tagged @smoke, one browser, fourteen minutes on the pull request. The team merges and then reads the result, which means it is not a gate any more. I do not want to…
asked by Tanvi Bhatt126 ·A typo in -t made CI green for three weeks — why does matching nothing exit 0?
Someone renamed our pull-request step and dropped a letter: Three weeks of green pull requests. Nothing ran. The output even says so if you scroll up: A green tick under a…
asked by Bruno Teixeira694 ·My page object needs a signed-in session before goto() — where does the login belong?
Every one of our scenarios opens with the same three steps: go to login, fill two fields, submit. Then the real scenario starts. It is slow — about four seconds a scenario across…
asked by Helle Borg352 ·First UI run: five scenarios fail on the sign-in page, but the API layer was green
Working through the workshop on my own checkout of the sample app. The API layer passed straight away. Then the browser: Five failures, and as far as I can tell they are all the…
asked by Yusuf Demir201 ·How do you actually decide between @smoke, @regression and @sanity?
Lint makes me pick exactly one, which is fine, but the three names do not tell me much and everyone on the team has a different theory. Right now roughly everything is…
asked by Sunita Kale982 ·What should a pull request run, and what should wait for the nightly?
We have about 400 scenarios and currently every pull request runs all of them on three browsers. It takes 35 minutes and people merge without waiting, which makes the whole thing…
asked by Elsa Nyman333 ·Lint says "found @smoke @regression" — why can a scenario not be in both suites?
I tagged our happy-path checkout scenario with both suite tags because I want it in the pull-request smoke run and in the nightly regression run. Lint disagrees: Surely wanting a…
asked by Annika Solheim14,002 ·CI has been green for a week and I do not think the suite ran at all
Someone renamed a tag and our pipeline step is: Exit code 0, green check, everyone happy. I only looked because a bug walked into production that we have a scenario for.…
asked by Karine Moreau655 ·Do -l, -b and -t stack, or does the last one win?
I have been typing this without really knowing what it means: Three filters. Do they narrow each other down, or is one of them authoritative? Specifically: I have a scenario…
asked by Avery Hollis188 ·sdods run -l api finished in under two seconds — did it actually run anything?
I pointed the CLI at staging and asked for the API layer only: It printed nine passing scenarios and was finished before I had read the command back. No browser window, no window…
asked by Dinuka Perera412 ·