Questions tagged regression
The broad suite, usually nightly.
20 questions
Does --repeat-each 5 with --shard 1/4 repeat inside the shard or across shards?
Setting up a soak job: run the regression five times over, spread across four machines, and fail if anything is flaky. What I cannot work out is whether the repeats are part of…
asked by Marijn Kuiper18,840 ·Scenarios for a feature flag that is on in staging and off in production
Our new checkout is behind a build-time flag. On staging it is on, on the production build it is off, and the flag is baked into the bundle at build time so there is nothing to…
asked by Zeynep Arslan64 ·Is there a --since? Every pull request runs the entire suite
A one-line change to a CSS file runs 900 scenarios. Other tools I have used take a diff and work out which tests could possibly be affected. I have read the flag list twice and I…
asked by Paulo Mendes97 ·Quarantined scenarios still run in our release gate and still fail it
We quarantine flaky scenarios in the web UI while somebody fixes them. The dashboard shows them as quarantined, so I assumed that was the end of it. The release gate disagrees.…
asked by Ingrid Solberg241 ·Every scenario passed and the release gate still failed the job
Nothing failed. Zero. And the job is red, so the release is blocked and I am the one holding it. I can see the two flaky scenarios in the report and they both ended up passing.…
asked by Koen Vermeulen309 ·Two Examples blocks, one @smoke and one @regression — lint says found @smoke @regression
I wanted one outline with a small set of rows in smoke and the long tail in regression: Both tables are tagged once. Is this rule just not aware of Examples-level tags?
asked by Koen Vermeulen309 ·Tagged a scenario @a11y and it went green in four seconds — how do I tell if axe ran?
Added @a11y to two existing scenarios expecting them to get slower and probably to fail, because the pages in question are not good. Both passed, in about the same time as…
asked by Koen Vermeulen309 ·Shard 3 finishes four minutes after shard 1 — is there anything smarter than i/n?
We run the regression across eight shards. The wall-clock time of the job is set by the slowest shard, and ours are consistently uneven — shard 3 carries two long visual features…
asked by Marijn Kuiper18,840 ·The API added a field — how do I re-record so the old entries stop coming back?
Backend renamed a field. I re-recorded: The scenario passes now, but the file grew instead of being replaced, and the old response with the old field name is still sitting in…
asked by Nikhil Sane487 ·Does -t "@regression and not @mock" replace the layer filter or add to it?
I want the nightly regression minus anything that runs against mocked network, so: It ran fewer scenarios than -t @regression on the same layer, which is what I wanted, but I…
asked by Fiona McAllister159 ·We renamed one button and 40 scenarios healed in the same run — now what?
Design renamed "Continue" to "Next" on the checkout stepper. Nothing failed. The nightly regression run is green and carries forty-one heal events, all with the same description,…
asked by Bea Lindqvist4,318 ·Is there a dry run? I want to see what -t and -m select before I wait ten minutes
Writing a new pipeline step and I keep guessing at the selection, running it, waiting, and finding out I was wrong. Twice now the wrong thing was "it selected nothing", which is…
asked by Helle Borg352 ·One scenario fails about one run in twenty — what do I do with it?
We have a checkout scenario that fails maybe one run in twenty. Different step each time, usually a timeout. Rerunning is always green, so nobody has ever debugged it, and by now…
asked by Chandra Pillai873 ·We set --retries 3 and the nightly went green. Why is everyone unhappy?
Genuine question, not a troll. Our nightly had six or seven failures a week, all different, all "it passes when I run it". We set retries to 3 and it has been green for nine…
asked by Ade Oyinlola419 ·How do I prove the checkout scenario is flaky instead of arguing about it?
One scenario fails maybe one run in six. Every time I raise it I am told it passed locally, and every time I run it locally it passes, so I am the person crying wolf about the…
asked by Owen Brackley144 ·How many workers on a two-core build box, and can I stop at the first failure?
Two questions from the same afternoon of tuning. Our agents have two cores. I have seen -w 8 in examples, which on our box makes the UI run slower and adds timeouts. What is the…
asked by Lars Vinter276 ·How do you actually decide between @smoke, @regression and @sanity?
Lint makes me pick exactly one, which is fine, but the three names do not tell me much and everyone on the team has a different theory. Right now roughly everything is…
asked by Sunita Kale982 ·-t @regression,@mock ran more scenarios, not fewer
I wanted the regression scenarios that are mocked, so I asked for both tags: It ran considerably more than either tag has on its own. I expected a comma to mean "and". What does…
asked by Gio Castellano521 ·What should a pull request run, and what should wait for the nightly?
We have about 400 scenarios and currently every pull request runs all of them on three browsers. It takes 35 minutes and people merge without waiting, which makes the whole thing…
asked by Elsa Nyman333 ·Lint says "found @smoke @regression" — why can a scenario not be in both suites?
I tagged our happy-path checkout scenario with both suite tags because I want it in the pull-request smoke run and in the nightly regression run. Lint disagrees: Surely wanting a…
asked by Annika Solheim14,002 ·