Questions tagged reporting
Run artefacts, HTML reports, traces and insights.
32 questions
Locator fragility table is empty in insights but heal report has plenty of rows
gives me eleven rows, two of them with double-digit occurrences. prints suite health and the scenario table, and the locators section is empty. Runs are being ingested — there is…
asked by Paulo Mendes97 ·sdods://screenshot resource returns No such screenshot for a run I can open in the UI
Asking the model to compare the before and after images of a failed step. It resolves the run fine — run_get_scenario comes back with the screenshot URIs — but reading one of…
asked by Zeynep Arslan64 ·A cleanup DELETE returned 404 and the scenario still passed — is that intended?
We register cleanups the way the guide shows: The records are not being deleted. Looking at the run detail, the cleanup ran and the request 404'd, and the scenario is green. Two…
asked by Zeynep Arslan64 ·Run directory not found after moving the artifacts root, exit 2
We moved the artifacts root off the workspace disk on our agents: The run is fine and the directory is there with everything in it. The step afterwards that publishes the run to…
asked by Zeynep Arslan64 ·Can I get LCP out of a run as a recorded number, given budgets are not enforced?
I have read the workshop chapter, so I know @perf does not gate anything today and I am not asking for the gate. What I want is the measurement recorded per run so I can watch it…
asked by Zeynep Arslan64 ·heal_events, heal_locator_stats and sdods heal report — which do I use for what?
Building a small internal dashboard on top of the MCP server. There appear to be three ways to get at healing data and I cannot tell from the tool descriptions where the…
asked by Zeynep Arslan64 ·No step for downloads — how do I assert that an export produced a real CSV?
We have about a dozen export buttons. The most I can assert from a feature file today is that a toast appeared, which passed happily for two weeks while the export was returning…
asked by Karine Moreau655 ·Adding Allure without losing the SDODS dashboard and the ndjson
Our QA lead wants Allure because that is what the other teams publish. I am happy to produce it, but the last time I touched a reporter list in another tool I lost everything…
asked by Paulo Mendes97 ·I quarantined a flaky scenario and it still reds the gate
Followed the flaky triage advice and quarantined the worst offender: It shows as quarantined on the dashboard. It also still fails the nightly, and the gate is still red, which…
asked by Renata Kohl608 ·Where do I actually look at expected vs actual after a @visual failure on CI?
The nightly went red on a baseline scenario. I can see the scenario failed and I can read the pixel count in the log, which tells me nothing about whether somebody changed a…
asked by Ingrid Solberg241 ·Getting the HTML report out of the Docker image we run the suite in
We run the suite inside ghcr.io/siri1410/sdods-server on a self-hosted agent. The run works and the exit code is right, then the container goes away and takes the report with it.…
asked by Ingrid Solberg241 ·Job shows exit 130 — did the suite fail or not?
Evaluating SDODS against two other tools, so forgive the basic question. A nightly ended with exit code 130 and there is no failing scenario in the report — 212 passed, 0 failed,…
asked by Devon Marsh183 ·Insights say no runs, but we have a week of nightly runs on disk
Fine, so I ran compute: We have run this suite every night for a week and every run is sitting in .sdods/runs with its summary and its ndjson. Why does counting them need a…
asked by Rasha Halabi572 ·Docs call the insights CLI a placeholder but it clearly does something
The known limitations table says: It does not behave like a placeholder: wrote .sdods/insights/demo-shop.json, printed suite health with a trend, a scenarios-by-flakiness table…
asked by Koen Vermeulen309 ·Where do the generated spec files live? Git does not see them and neither do I
I understand roughly that the Gherkin gets turned into runner specs before a run. I cannot find them. Nothing appears in my working tree during a run, git status stays clean, and…
asked by Bruno Teixeira694 ·Getting run results onto a team dashboard without scraping the CLI
Management wants a pass-rate trend on the wall screen. Today somebody screenshots the console output on Monday morning. I would like to do this properly but I cannot tell which…
asked by Tanvi Bhatt126 ·Runs are 900MB of PNGs — can I keep the baseline check and stop capturing every step?
Nightly regression writes a before and an after image for every UI step, which is wonderful when something fails and enormous every other night. The artefact bucket is the…
asked by Sergio Alcaraz415 ·What writes error-context.md next to a failure, and can I use it?
Poking through a failed run I found this: The markdown file starts with the headings Instructions, Test info, Error details and Page snapshot, and the page snapshot is a full…
asked by Fiona McAllister159 ·Server ingest takes the ndjson but the run viewer has no screenshots
Our CI agents have no database credentials, so we ingest to the server with a scoped token instead. Messages go up fine. Screenshots do not. First attempt was my own fault: With…
asked by Bruno Teixeira694 ·Trace links in the web UI give a 404 from /trace/ — is it my deployment?
Running sdods serve behind our reverse proxy. The run detail page is fine, screenshots load, the API panels load, and then the trace link goes to: Every other static path under…
asked by Nikhil Sane487 ·sdods heal report says No run found. but the suite ran two minutes ago
Exit code 2. The run directory is there and full of files — I can open heal.jsonl inside it.
asked by Nikhil Sane487 ·What do the columns of sdods heal report actually mean?
I get a table with description, original, occurrences, succeeded, strategy and suggested. Most of it I can guess. Two I cannot:
asked by Helle Borg352 ·@jira:demo12 must look like PROJ-123 — what patterns do the issue tags accept?
Two errors from the same commit: I get that the second one wants a number. What is wrong with demo12? Our board key really is lowercase in the URL.
asked by Helle Borg352 ·One scenario fails about one run in twenty — what do I do with it?
We have a checkout scenario that fails maybe one run in twenty. Different step each time, usually a timeout. Rerunning is always green, so nobody has ever debugged it, and by now…
asked by Chandra Pillai873 ·What exactly is "healed": 1 counting in summary.json?
Run went green and the summary said: Green is green, but I would like to know what got healed before I sign the release off. Is healed counting scenarios, steps or locators, and…
asked by Renata Kohl608 ·We set --retries 3 and the nightly went green. Why is everyone unhappy?
Genuine question, not a troll. Our nightly had six or seven failures a week, all different, all "it passes when I run it". We set retries to 3 and it has been green for nine…
asked by Ade Oyinlola419 ·How long should we keep .sdods/runs around? It is 40 GB now
Nobody set a retention policy when we turned this on and the build agent is now 40 GB of .sdods/runs, almost all of it step screenshots. I do not want to rm -rf blindly because…
asked by Owen Brackley144 ·Reading summary.json in a shell script — which keys are stable?
I want the CI job to post totals to Slack without scraping console output. .sdods/runs/<id>/summary.json looks like exactly what I need: Before I build on it: is this file part…
asked by Gio Castellano521 ·What is the least painful way to hand a trace to a developer?
A checkout scenario fails only on the CI machine. I can open the trace with sdods trace --last and it is clear enough what happened, but the developer who has to fix it does not…
asked by Elsa Nyman333 ·Three shards, three HTML reports — how do I get one report out of a matrix build?
The nightly runs on three machines: Every machine mints its own run id, so I get three artefact directories, three HTML reports and three rows in the dashboard for what is one…
asked by Vikram Rajagopal1,240 ·sdods report prints (missing) for the html report after a CI run
Our CI job runs the suite and archives the run, someone downloads the archive and opens it locally. Locally, straight after a run, everything is there. On the downloaded copy: So…
asked by Tobi Schrader97 ·Where does a run actually write its files? I cannot find the report
First real run finished and the console printed the totals, but I have no idea where anything landed. I was expecting a reports/ folder next to features/. There is a .sdods…
asked by Hsu Wei-lin15,230 ·