Questions tagged ci
Running suites on a build server: sharding, artefacts and gates.
74 questions
Installing onto a machine with no outbound internet at all
Air-gapped test lab. No proxy to configure, no mirror to point at — the machines genuinely have no route out, and files arrive on a reviewed share. What I am trying to work out…
asked by Jide Okonkwo11,745 ·Linting only the staged feature files in a pre-commit hook — is --json stable enough?
Trying to keep the hook under a second on a repo with about four hundred feature files, so linting the whole project every commit is out. This is what I have: Two things I cannot…
asked by Ingrid Solberg241 ·Does --repeat-each 5 with --shard 1/4 repeat inside the shard or across shards?
Setting up a soak job: run the regression five times over, spread across four machines, and fail if anything is flaky. What I cannot work out is whether the repeats are part of…
asked by Marijn Kuiper18,840 ·Can I get LCP out of a run as a recorded number, given budgets are not enforced?
I have read the workshop chapter, so I know @perf does not gate anything today and I am not asking for the gate. What I want is the measurement recorded per run so I can watch it…
asked by Zeynep Arslan64 ·Can I build the server image myself and push it to our own registry?
Our cluster only pulls from an internal registry, so pulling from a public one is not an option regardless of how convenient it is. Is building the image ourselves supported, or…
asked by Zeynep Arslan64 ·Scenarios for a feature flag that is on in staging and off in production
Our new checkout is behind a build-time flag. On staging it is on, on the production build it is off, and the flag is baked into the bundle at build time so there is nothing to…
asked by Zeynep Arslan64 ·Every PR runs all 190 specs — is there a --since for sdods run?
PR checks take 40 minutes because we run the whole suite on every push, and a typical PR touches one module. I was expecting something like but that is not a flag. There is…
asked by Thom Vasseur5,470 ·Can I run a suite with the Docker image, or is it only the server?
The image is described as the server and the web UI. Our CI does not need a UI, it needs to run the API layer and fail the build. Is the image the wrong tool for that, or does it…
asked by Bruno Teixeira694 ·I quarantined a flaky scenario and it still reds the gate
Followed the flaky triage advice and quarantined the worst offender: It shows as quarantined on the dashboard. It also still fails the nightly, and the gate is still red, which…
asked by Renata Kohl608 ·My @env:local scenarios ran against staging — does @env: actually skip anything?
The tags table says @env:<name> "restricts a scenario to one environment", and sdods lint is happy with the tag. But: ran all 41 of our @env:local scenarios against staging. Some…
asked by Lars Vinter276 ·--strict on sdods run did not turn warnings into failures like it does on sdods lint
We wanted the nightly to fail on lint warnings, so somebody moved the flag onto the run: It ran, it passed, and the eleven warnings we were trying to catch are still there. On…
asked by Paulo Mendes97 ·295 API scenarios fail with QUOTA_EXCEEDED: Exceeded quota for verifying passwords
Ran the API and hybrid layers, single worker, one browser. 1,152 tests. 295 of the failures are this one message: The credentials are fine. If I run any one of those scenarios on…
asked by Ade Oyinlola419 ·Is there a --since? Every pull request runs the entire suite
A one-line change to a CSS file runs 900 scenarios. Other tools I have used take a diff and work out which tests could possibly be affected. I have read the flag list twice and I…
asked by Paulo Mendes97 ·What do the installer exit codes mean? They do not match the CLI ones
I want our provisioning script to react differently depending on why the install failed — retry a download, but fail the build for a bad flag. I found the CLI exit code table and…
asked by Nikhil Sane487 ·335 of 390 scenarios failed at -w 4 with the user pool exhausted, and doctor is all green
Full regression at four workers. 55 passed, 335 failed, and nearly all of the failures are this: The passes are all scenarios that need no user, or need @user:admin, which…
asked by Chandra Pillai873 ·Where do I actually look at expected vs actual after a @visual failure on CI?
The nightly went red on a baseline scenario. I can see the scenario failed and I can read the pixel count in the log, which tells me nothing about whether somebody changed a…
asked by Ingrid Solberg241 ·Getting the HTML report out of the Docker image we run the suite in
We run the suite inside ghcr.io/siri1410/sdods-server on a self-hosted agent. The run works and the exit code is right, then the container goes away and takes the report with it.…
asked by Ingrid Solberg241 ·"@sdods/cli is not available on the npm registry" behind an internal mirror
Our build network can only reach an internal registry mirror. The installer stops before it downloads anything: Exit code 7. The package resolves fine from my laptop, so I assume…
asked by Ify Adeyemi6,584 ·Should each tester run sdods mcp locally, or should we host one server for the team?
Eight testers, one repo, one staging environment. Right now everybody has stdio in their editor and their own checkout. It works, but nobody can ask about a run somebody else…
asked by Ingrid Solberg241 ·Quarantined scenarios still run in our release gate and still fail it
We quarantine flaky scenarios in the web UI while somebody fixes them. The dashboard shows them as quarantined, so I assumed that was the end of it. The release gate disagrees.…
asked by Ingrid Solberg241 ·One viewer account, twelve read-only scenarios, and the suite behaves as if -w 1
A partner gives us exactly one viewer account and will not create more — it is their system, their policy, and no amount of asking has moved it. Twelve scenarios carry…
asked by Rasha Halabi572 ·Can I turn healing off for one scenario without touching the whole project?
We have one scenario whose entire job is to assert that the design system has not moved the primary action button. Healing makes it unfalsifiable — it heals to the role and…
asked by Bea Lindqvist4,318 ·A renamed tag left our smoke job passing with zero tests for three weeks
Three weeks. The only trace, in a log nobody opens: I understand from an older thread here that an empty selection exits 0 on purpose so a matrix leg with nothing to do stays…
asked by Tanvi Bhatt126 ·Catching broken configuration in the pull request instead of ten minutes into the suite
Twice this month a pull request that only touched an environment file went green on review and then took the nightly down. Both times it was something a schema check would have…
asked by Anouk de Vries238 ·Job shows exit 130 — did the suite fail or not?
Evaluating SDODS against two other tools, so forgive the basic question. A nightly ended with exit code 130 and there is no failing scenario in the report — 212 passed, 0 failed,…
asked by Devon Marsh183 ·
Earlier (49)
- SDODS_UI_BASE_URL in the build image overrides -e staging and there seems to be no flag to beat it0 answers · 2025
- Running the smoke suite on a plane — what do I need so nothing touches the network?4 answers✓ · 2025
- Every scenario passed and the release gate still failed the job2 answers✓ · 2025
- Getting run results onto a team dashboard without scraping the CLI3 answers · 2025
- Our @smoke check takes fourteen minutes and people have stopped waiting for it4 answers✓ · 2025
- sdods run refuses to start with LINT_FAILED — can I run anyway while I fix the tags?3 answers✓ · 2025
- --project-matrix or three -b flags? What is the difference in practice?3 answers✓ · 2025
- Node v20 is too old — can the installer sort that out itself?3 answers✓ · 2024
- Shard 3 finishes four minutes after shard 1 — is there anything smarter than i/n?0 answers · 2024
- perf.budgets validates and resolves, and a nine-second page still passes. What am I missing?2 answers✓ · 2024
- Team-wide setup for keeping credentials out of the repo — what actually works day to day?4 answers · 2024
- Some browsers are not installed. — fresh agent, first run of the day2 answers✓ · 2024
- Cancelled a run with ctrl-c and now every pool account looks leased3 answers✓ · 2024
- Recorded with four workers and half the requests are missing from the file1 answer✓ · 2024
- A typo in -t made CI green for three weeks — why does matching nothing exit 0?4 answers✓ · 2024
- Invalid shard "1-3", expected "<current>/<total>" like 1/31 answer✓ · 2024
- Server ingest takes the ndjson but the run viewer has no screenshots2 answers✓ · 2024
- Do you commit baseline PNGs into the repo, or is this what git-lfs is for?4 answers✓ · 2024
- Process "pr-chek" is not defined for project demo-shop — where do processes come from?2 answers✓ · 2024
- --headed on the build agent just sits there until the job times out1 answer✓ · 2024
- Pipeline exits 3 and no test ran — the report is empty3 answers✓ · 2024
- Why is a completely made-up tag only a warning while @jira:demo12 fails the build?4 answers✓ · 2024
- Trace links in the web UI give a 404 from /trace/ — is it my deployment?2 answers✓ · 2024
- Is there a dry run? I want to see what -t and -m select before I wait ten minutes2 answers✓ · 2023
- Is it safe to leave self-healing enabled in CI, or is that cheating?2 answers✓ · 2023
- One scenario fails about one run in twenty — what do I do with it?4 answers✓ · 2023
- Green on my machine, red on the runner — it was replaying a HAR the whole time2 answers✓ · 2023
- Every API scenario fails at once with a connection error — where do I start?3 answers✓ · 2023
- CI job exits 2 with no test results and an empty report — what is code 2?2 answers✓ · 2023
- We set --retries 3 and the nightly went green. Why is everyone unhappy?3 answers✓ · 2023
- Suite went from six minutes to nineteen after we enabled healing3 answers✓ · 2023
- Should HAR files go into git? Ours are nearly a megabyte each3 answers✓ · 2023
- How do I prove the checkout scenario is flaky instead of arguing about it?2 answers✓ · 2023
- How many workers on a two-core build box, and can I stop at the first failure?2 answers · 2022
- If I pass --process nightly-regression and -t @smoke, which one wins?1 answer✓ · 2022
- How long should we keep .sdods/runs around? It is 40 GB now2 answers✓ · 2022
- Refusing to install as root — our build agent only has a root account2 answers✓ · 2022
- All 4 user(s) with role "standard" are leased — passes locally, four failures in CI4 answers✓ · 2022
- @visual is green locally and reported flaky on CI — attempt 1 fails, attempt 2 passes3 answers✓ · 2022
- Reading summary.json in a shell script — which keys are stable?2 answers✓ · 2022
- Everyone on the team runs a slightly different command. Can I name one?2 answers✓ · 2022
- Value in .env.staging is ignored, an old exported variable keeps winning2 answers✓ · 2022
- What should a pull request run, and what should wait for the nightly?2 answers · 2021
- I do not want healing to rescue a broken locator — I want the test to fail4 answers✓ · 2021
- sdods record on our Linux build agent: codegen exited with code 1 and produced no spec3 answers✓ · 2021
- Three shards, three HTML reports — how do I get one report out of a matrix build?3 answers✓ · 2021
- sdods report prints (missing) for the html report after a CI run2 answers✓ · 2021
- Our CI step passes even when lint fails — checking for exit code 1 was wrong?1 answer✓ · 2021
- CI has been green for a week and I do not think the suite ran at all3 answers✓ · 2021