Example threads. These questions and answers were written by the SDODS team to document real problems and how to fix them. The people named are illustrative, not real users. Questions asked on this site are marked new.
120 scenarios tagged @smoke, one browser, fourteen minutes on the pull request. The team merges and then reads the result, which means it is not a gate any more.
I do not want to just delete scenarios. Where does the time actually go in a suite like this, and what are the levers in SDODS specifically?
Accepted answer
Measure first — summary.json has per-scenario durations and the dashboard has the duration chart — but the costs are nearly always in this order:
auth.storageState: true and use @user:<role>. Login then happens once per pool user and every scenario reuses the state. On a 120-scenario suite this alone is usually minutes.@api and never open a browser.--strict, so the run does not wait on somebody else's sandbox.step policy for @regression and use onlyOnFailure on slow environments.-w and --shard are there. @mode:serial should be rare and deliberate.And the anti-lever: retries. If the run is slow because things fail and retry, raising retries makes it slower and hides the reason.
Once the per-scenario waste is out, buy the rest with machines. We shard the pull request check six ways with a shared --run-id and merge the reports; 4,000 scenarios, about six minutes wall clock.
Sharding is not a substitute for the first answer though. Six shards of a suite that logs in 120 times is still 120 logins, just spread out.
If any part of your smoke path touches a payment or identity sandbox, HAR replay is the single biggest win available to you and it is not close. Record once with --har-update, then run the gate with --har-replay --strict. Ours went from four minutes of waiting on a third party to nothing, and stopped failing on their maintenance windows as a bonus.
Before you optimise anything, look at the slowest scenarios in insights. Twice now the answer here has been one scenario with a hard wait in a step, and everybody had been arguing about workers.
Wrap commands and output in triple backticks and they will render as a code block.
Checking whether you’re signed in…
Have a different question? Ask it.