What writes error-context.md next to a failure, and can I use it?

Asked 25 Nov 20244,980 views3 answers

17votes

Poking through a failed run I found this:

text

.sdods/runs/019362f1-.../runner-output/demo-shop-ui-chromium-Checkout-with-a-saved-card/
  error-context.md
  test-failed-1.png
  trace.zip

The markdown file starts with the headings Instructions, Test info, Error details and Page snapshot, and the page snapshot is a full accessibility tree of the page at the moment it died. Nobody on the team knew it existed.

Is it stable, and is there any reason not to paste it straight into a bug report?

reporting cli

asked by Fiona McAllister159 · 25 Nov 2024

3 answers

21votes✓
Accepted answer
It is the failure context, written per failing test into that test's output directory, and it is the most useful file in a failed run for anybody who was not watching it happen. The four sections are always in that order:
- Instructions — a preamble aimed at whoever, or whatever, is going to fix the test
- Test info — which test, which file
- Error details — the error and the failing call
- Page snapshot — the accessibility tree of the page at the moment of failure
The page snapshot is the reason to paste it instead of a screenshot: it is text, it diffs between two runs, and it names roles and accessible names, so "the button is called Place order now, not Place Order" falls straight out of it. A screenshot makes somebody squint at pixels to reach the same conclusion.
One caution. It is a snapshot of the page, so it contains whatever was on screen — order numbers, the seeded user's email, anything a fixture put there. Read it before it goes into a tracker other people can see.
answered by Anouk de Vries238 · 25 Nov 2024
13votes
Adding the path detail, because it moved. Failure context and traces live under runner-output/ in current runs and under pw-output/ in older ones, the same way the HTML report is html-report/ now and was playwright-report/ before the rename. Both names are still read when a run is ingested, so old run directories keep working.
A script that hardcodes one of them, though, will quietly find nothing across half your history and report zero failures with a straight face. Let the CLI find the files:
bash
```
sdods trace --run 019362f1-8b44-7e02-a0c7-5d19be302f61 --list
```
answered by Ify Adeyemi6,584 · 27 Nov 2024
6votes
We append it to the Jira issue body. createIssueOnFailure already gets you the run link and the error; adding the Error details and Page snapshot sections means the developer has the whole story without opening SDODS at all.
Acceptance went up noticeably once we stopped sending a screenshot and a stack trace and started sending something they could search.
answered by Sam Orbison9,420 · 3 Dec 2024

Have a different question? Ask it.

What writes error-context.md next to a failure, and can I use it?

3 answers

Know the answer?

Related