120 scenarios tagged @smoke, one browser, fourteen minutes on the pull request. The team merges and then reads the result, which means it is not a gate any more.
I do not want to just delete scenarios. Where does the time actually go in a suite like this, and what are the levers in SDODS specifically?
Accepted answer
Measure first — summary.json has per-scenario durations and the dashboard has the duration chart — but the costs are nearly always in this order:
auth.storageState: true and use @user:<role>. Login then happens once per pool user and every scenario reuses the state. On a 120-scenario suite this alone is usually minutes.@api and never open a browser.--strict, so the run does not wait on somebody else's sandbox.step policy for @regression and use onlyOnFailure on slow environments.-w and --shard are there. @mode:serial should be rare and deliberate.And the anti-lever: retries. If the run is slow because things fail and retry, raising retries makes it slower and hides the reason.
Once the per-scenario waste is out, buy the rest with machines. We shard the pull request check six ways with a shared --run-id and merge the reports; 4,000 scenarios, about six minutes wall clock.
Sharding is not a substitute for the first answer though. Six shards of a suite that logs in 120 times is still 120 logins, just spread out.
If any part of your smoke path touches a payment or identity sandbox, HAR replay is the single biggest win available to you and it is not close. Record once with --har-update, then run the gate with --har-replay --strict. Ours went from four minutes of waiting on a third party to nothing, and stopped failing on their maintenance windows as a bonus.
Before you optimise anything, look at the slowest scenarios in insights. Twice now the answer here has been one scenario with a hard wait in a step, and everybody had been arguing about workers.
Have the same question?
Ask it with the command you ran and the output you got, and it will be answered here.
Ask a question