The nightly went red on a baseline scenario. I can see the scenario failed and I can read the pixel count in the log, which tells me nothing about whether somebody changed a button or a font.
Is there a view that puts the committed baseline and the actual capture next to each other, or am I meant to download the artefacts and open them in an image viewer?
Accepted answer
There is a view. Open the run, go to the Scenarios tab, expand the scenario, and the step that ran the baseline assertion carries a block headed "Visual baseline" with the name you gave it. Inside that block the expected and the actual image are loaded together, and it opens in diff mode by default — there is also a slider, an overlay and plain side-by-side if the diff is too noisy to read.
Locally the same thing without the server:
sdods report --last --openThe slider is the one I actually use. A font change moves every glyph edge slightly and looks like noise in a diff; drag the slider and it is obvious in a second.
Caveat for the CI half of the question: those images only reach the server if the run is ingested on the machine that produced them. The ingest route accepts run.json, NDJSON and runner JSON — it does not accept an artefacts archive yet, so uploading a tarball from a later job gets you the rows and none of the pictures.
Two ways round it:
--ingest on the CI run itself, so it ingests from the runner while the files are still there.sdods report ingest --artifacts-dir .sdods/runs as a later step in the same job.Listed under Known limitations as "Server ingest".
Have the same question?
Ask it with the command you ran and the output you got, and it will be answered here.
Ask a question