Lena Hartwig
Flaky triage. Keeps a spreadsheet of every quarantined scenario.
Illustrative user
Example threads. These questions and answers were written by the SDODS team to document real problems and how to fix them. The people named are illustrative, not real users. Questions asked on this site are marked new.
Asked (1)
Example threads. These questions and answers were written by the SDODS team to document real problems and how to fix them. The people named are illustrative, not real users. Questions asked on this site are marked new.
Scenario tagged @har:products runs green but har/staging is still empty
I added the tag and ran the suite. It passes, but nothing lands in projects/demo-shop/har/staging/. The only odd thing in the output was a yellow line I scrolled past: Do I have…
asked by Lena Hartwig
Answered (26)
Example threads. These questions and answers were written by the SDODS team to document real problems and how to fix them. The people named are illustrative, not real users. Questions asked on this site are marked new.
Adding Allure without losing the SDODS dashboard and the ndjson
Our QA lead wants Allure because that is what the other teams publish. I am happy to produce it, but the last time I touched a reporter list in another tool I lost everything…
asked by Paulo MendesCan I turn healing off for one scenario without touching the whole project?
We have one scenario whose entire job is to assert that the design system has not moved the primary action button. Healing makes it unfalsifiable — it heals to the role and…
asked by Bea LindqvistOur nth-child selectors break every sprint and healing never rescues them
Inherited suite, roughly two hundred locators in this shape: Every time the results list gains a badge or a promo row, a chunk of them break, and they break with HEAL_FAILED…
asked by Tanvi BhattOur @smoke check takes fourteen minutes and people have stopped waiting for it
120 scenarios tagged @smoke, one browser, fourteen minutes on the pull request. The team merges and then reads the result, which means it is not a gate any more. I do not want to…
asked by Tanvi Bhattsdods run refuses to start with LINT_FAILED — can I run anyway while I fix the tags?
Mid-refactor, half my tags are wrong, and I want to run the six scenarios that are fine to check a page object change. Instead: Nothing ran. The broken scenarios are not even in…
asked by Tanvi BhattThe healer proposed a CSS locator when there is a perfectly good label — reject it?
Scenario broke, ran the healer, got a proposal. The patch swaps a page-object locator for a class chain: The input has a visible label reading "Promo code". So there is an…
asked by Anouk de VriesHEAL_FAILED: nothing scored above 0.6 and testId probed at 0.00
The context is: What I cannot work out is testid=0.00. It is the same test id the primary uses, so of course it found nothing — but then why probe it at all? And where does 0.42…
asked by Anouk de VriesWe renamed one button and 40 scenarios healed in the same run — now what?
Design renamed "Continue" to "Next" on the checkout stepper. Nothing failed. The nightly regression run is green and carries forty-one heal events, all with the same description,…
Why is a completely made-up tag only a warning while @jira:demo12 fails the build?
Trying to understand the severity model, because it looks backwards to me. @foo:bar is total nonsense and it only warns. @jira:demo12 is an obvious typo of a real thing and it…
asked by Nikhil SaneOne scenario fails about one run in twenty — what do I do with it?
We have a checkout scenario that fails maybe one run in twenty. Different step each time, usually a timeout. Rerunning is always green, so nobody has ever debugged it, and by now…
asked by Chandra PillaiWhere exactly does the "# title-format:" comment go? Lint still warns after I added one
Trying to clear this: I added the comment and the warning did not move. My file: Does it have to be in a particular place?
asked by Chandra PillaitestIdAttribute is data-test, so why does the heal suggestion say getByTestId?
Our app has used data-test since long before we adopted SDODS. I set it in the project file: Heal report suggests getByTestId('cart-badge') for a locator that healed. I pasted…
asked by Yusuf DemirWe set --retries 3 and the nightly went green. Why is everyone unhappy?
Genuine question, not a troll. Our nightly had six or seven failures a week, all different, all "it passes when I run it". We set retries to 3 and it has been green for nine…
asked by Ade OyinlolaHow do I prove the checkout scenario is flaky instead of arguing about it?
One scenario fails maybe one run in six. Every time I raise it I am told it passed locally, and every time I run it locally it passes, so I am the person crying wolf about the…
asked by Owen Brackley@env:qa rejected with "is not in envs.available (local, staging)" — we do have a qa env
We added projects/shop/envs/qa.yaml last week and it works — sdods run -p shop -e qa -l api runs against it. But the moment I pin a scenario to it: The file exists. What else is…
asked by Ade OyinlolaCan I put an XPath in the css fallback list of a heal context?
Our app renders a summary table where the only stable thing about a row is its position relative to a header cell. The reviewer agent keeps flagging the selector, so I was going…
How do you actually decide between @smoke, @regression and @sanity?
Lint makes me pick exactly one, which is fine, but the three names do not tell me much and everyone on the team has a different theory. Right now roughly everything is…
asked by Sunita KaleHow do you assert on a row in a table without writing a CSS chain?
Orders table, and the scenario I want reads: What I have in the page object is this, which works until someone sorts the table or the row moves to page two: Is there a…
asked by Sunita KaleReading summary.json in a shell script — which keys are stable?
I want the CI job to post totals to Slack without scraping console output. .sdods/runs/<id>/summary.json looks like exactly what I need: Before I build on it: is this file part…
asked by Gio Castellano"@user:admin is not a declared role (standard, locked_out)" — where is that list?
Added an admin-only scenario: We do have an admin account in the users CSV, with role set to admin. So why does lint not see it?
asked by Sam OrbisonWhy does healing never kick in for expectHidden()?
Every positive assertion in our page object heals fine. This one never does: assert is in heal.actions, the context has a role, a name and a test id. The step fails with an…
asked by Gio CastellanoNew folder under projects/ is ignored completely — no error, just not there
Made projects/billing/ by hand with an envs/local.yaml in it. sdods project list shows demo-shop, rwa-bank and sdods-web and no billing. No warning either. Then when I point at…
asked by Elsa NymanI do not want healing to rescue a broken locator — I want the test to fail
Possibly an unpopular opinion, but: if a developer renames a button and my test still passes, my test did not test anything. Yesterday a whole feature shipped with the wrong…
asked by Vikram RajagopalWhat is the least painful way to hand a trace to a developer?
A checkout scenario fails only on the CI machine. I can open the trace with sdods trace --last and it is clear enough what happened, but the developer who has to fix it does not…
asked by Elsa NymanCan a scenario be tagged @ui and @api at once? Lint says "found @hybrid @ui @api"
The scenario seeds a record over the API and then checks it renders in the browser, so it genuinely is both. I tagged it as both and got: Three? I only wrote two. The @hybrid is…
asked by Vikram RajagopalScenario Outline with 20 Examples rows, or a CSV dataset? What do reviewers expect here
We have a shipping tax table: twenty countries, each with a rate and an expected label on the confirmation page. I wrote it as a Scenario Outline and the feature file is now a…
asked by Jide Okonkwo