Marijn Kuiper
Runs a 4,000-scenario suite on a build farm and has opinions about shards.
Regular · 18,840 reputation · here since 21 Sept 2020
Asked (4)
Does --repeat-each 5 with --shard 1/4 repeat inside the shard or across shards?
Setting up a soak job: run the regression five times over, spread across four machines, and fail if anything is flaky. What I cannot work out is whether the repeats are part of…
asked by Marijn Kuiper18,840 ·Shard 3 finishes four minutes after shard 1 — is there anything smarter than i/n?
We run the regression across eight shards. The wall-clock time of the job is set by the slowest shard, and ours are consistently uneven — shard 3 carries two long visual features…
asked by Marijn Kuiper18,840 ·sdods lint: "Scenario must carry exactly one layer tag" on every scenario I have
Ported a feature file over from our old cucumber project and lint rejects all of it. The Gherkin is valid — it parses, the steps exist. What is a "layer tag", and does it belong…
asked by Marijn Kuiper18,840 ·Where is a page object supposed to live? Mine is never picked up
I put my page object in projects/demo-shop/support/pages/LoginPage.ts because that is how our old framework was laid out. It compiles, the decorators are on the methods, and the…
asked by Marijn Kuiper18,840 ·
Answered (26)
macOS says "SDODS is damaged and can't be opened" after downloading the app
Downloaded the desktop app, dragged it to Applications, double-clicked, and: Downloaded it twice in case something was truncated. Same message. Is the download actually broken or…
asked by Gio Castellano521 ·Every PR runs all 190 specs — is there a --since for sdods run?
PR checks take 40 minutes because we run the whole suite on every push, and a typical PR touches one module. I was expecting something like but that is not a flag. There is…
asked by Thom Vasseur5,470 ·Is there a --since? Every pull request runs the entire suite
A one-line change to a CSS file runs 900 scenarios. Other tools I have used take a diff and work out which tests could possibly be affected. I have read the flag list twice and I…
asked by Paulo Mendes97 ·335 of 390 scenarios failed at -w 4 with the user pool exhausted, and doctor is all green
Full regression at four workers. 55 passed, 335 failed, and nearly all of the failures are this: The passes are all scenarios that need no user, or need @user:admin, which…
asked by Chandra Pillai873 ·brew install asks me to trust the SDODS tap first — is that expected?
Ran the documented command: Homebrew stops and asks me to confirm I trust the tap, and prints a brew trust command to run. I have installed plenty of things with brew and not…
asked by Sergio Alcaraz415 ·I use no authentication still sends the bearer token — my 401 assertion gets a 200
Deny-direction scenario, as plain as it gets: The endpoint does deny anonymous callers — I checked with curl. Our envs/staging.yaml has: and if I remove that block the scenario…
asked by Gio Castellano521 ·Every generated API scenario hits the UI origin — ui and api baseUrl came out identical
Fresh project from analyze. The starter @api features all fail with HTML where JSON should be, and the reason is in the env file: Our API is at /api on the same origin. I…
asked by Yusuf Demir201 ·sdods analyze invented 18 page routes out of directories that happen to be called pages
We are App Router only. No pages/ directory anywhere near the router. But: comes back with a routes table full of things that do not exist: Eighteen of them. The same paths also…
asked by Sergio Alcaraz415 ·Our @smoke check takes fourteen minutes and people have stopped waiting for it
120 scenarios tagged @smoke, one browser, fourteen minutes on the pull request. The team merges and then reads the result, which means it is not a gate any more. I do not want to…
asked by Tanvi Bhatt126 ·Do you commit baseline PNGs into the repo, or is this what git-lfs is for?
We are about to turn @visual on for real — currently one scenario, about to be fifteen, on two browsers, and eventually two platforms once CI has its own baselines. Before I…
asked by Fiona McAllister159 ·Is there a dry run? I want to see what -t and -m select before I wait ten minutes
Writing a new pipeline step and I keep guessing at the selection, running it, waiting, and finding out I was wrong. Twice now the wrong thing was "it selected nothing", which is…
asked by Helle Borg352 ·We set --retries 3 and the nightly went green. Why is everyone unhappy?
Genuine question, not a troll. Our nightly had six or seven failures a week, all different, all "it passes when I run it". We set retries to 3 and it has been green for nine…
asked by Ade Oyinlola419 ·auth list says the cached state is fresh but the recording opens logged out
Starting a recording with --user standard drops me on the sign-in page, but the cache insists it has a good state: Eighteen minutes old, and we never touched auth.maxAgeMinutes,…
asked by Renata Kohl608 ·How many workers on a two-core build box, and can I stop at the first failure?
Two questions from the same afternoon of tuning. Our agents have two cores. I have seen -w 8 in examples, which on our box makes the UI run slower and adds timeouts. What is the…
asked by Lars Vinter276 ·Guide says baselines live under __screenshots__/<browser>/ — mine are in demo-shop--ui--chromium/darwin/
Screenshots and visual testing says: What is actually on disk after a run: Two extra segments and neither of them is just the browser. I want to write a cleanup script that…
asked by Gio Castellano521 ·Installer says "This is Windows" when I run it in Git Bash — what should I use instead?
I copied the macOS/Linux one-liner into Git Bash because that is where I live on this machine. It refuses immediately: I do have WSL. Which one is actually the right answer if I…
asked by Sam Orbison9,420 ·Warning: "Feature is not under a declared module directory" — but the file runs fine
I put a new file at features/smoke-checks.feature, straight under features/. It runs, it passes, but every lint gives me: It is only a warning so nothing is blocked, but it is…
asked by Gio Castellano521 ·sdods record on our Linux build agent: codegen exited with code 1 and produced no spec
We moved recording onto the same Linux build agent that runs the nightly suite so nobody has to install browsers locally. It dies before a browser ever appears. sdods run -p…
asked by Avery Hollis188 ·Three shards, three HTML reports — how do I get one report out of a matrix build?
The nightly runs on three machines: Every machine mints its own run id, so I get three artefact directories, three HTML reports and three rows in the dashboard for what is one…
asked by Vikram Rajagopal1,240 ·CI has been green for a week and I do not think the suite ran at all
Someone renamed a tag and our pipeline step is: Exit code 0, green check, everyone happy. I only looked because a bug walked into production that we have a scenario for.…
asked by Karine Moreau655 ·Do -l, -b and -t stack, or does the last one win?
I have been typing this without really knowing what it means: Three filters. Do they narrow each other down, or is one of them authoritative? Specifically: I have a scenario…
asked by Avery Hollis188 ·Coming from Selenium PageFactory — what replaces @FindBy and initElements?
We have a Java suite, maybe 300 tests, built the way everyone built them: a BasePage holding the driver and the waits, @FindBy(xpath = "//div[@class='cart']//button") fields, and…
asked by Dinuka Perera412 ·Where does a run actually write its files? I cannot find the report
First real run finished and the console printed the totals, but I have no idea where anything landed. I was expecting a reports/ folder next to features/. There is a .sdods…
asked by Hsu Wei-lin15,230 ·Is there a way to see what the install script will do before it does it?
Our security team will not let anyone pipe a URL into a shell on a company laptop, and honestly I agree with them. I want to install SDODS but I need to be able to show someone…
asked by Dinuka Perera412 ·DATASET_NOT_FOUND says my dataset is not declared, but the CSV is right there
Trying to drive the login scenarios from a CSV instead of an Examples table. The file is at projects/demo-shop/data/common/users.csv and previewing it works: Six rows, columns…
asked by Dinuka Perera412 ·Unresolved variable ${API_TOKEN} at "env.api.auth.token" — where is it supposed to look?
Added bearer auth to the staging environment file: Then: It dies before a single request goes out: The value is in a file next to the project yaml called env.staging. What am I…
asked by Dinuka Perera412 ·