@visual is green locally and reported flaky on CI — attempt 1 fails, attempt 2 passes

Example threads. These questions and answers were written by the SDODS team to document real problems and how to fix them. The people named are illustrative, not real users. Questions asked on this site are marked new.

Same commit, same command, two machines.

On my laptop:

bash

sdods run -p rwa-bank -e local -l ui -b chromium -t @visual

Green, every time. On the CI job (Ubuntu runner, project has retries: { ci: 2 }) the scenario is reported flaky. Attempt 1:

text

Error: A snapshot doesn't exist at projects/rwa-bank/features/__screenshots__/rwa-bank--ui--chromium/linux/signin.png, writing actual.

Attempt 2 passes and the run finishes green-with-a-flaky. Nothing in git changed between the attempts, obviously. This has happened on every nightly for a week and I do not understand what is unstable.

visual ci linux

asked by Sam Orbison

3 answers

Accepted answer
Nothing is unstable. Read the two paths next to each other:
text
```
projects/rwa-bank/features/__screenshots__/rwa-bank--ui--chromium/darwin/signin.png
projects/rwa-bank/features/__screenshots__/rwa-bank--ui--chromium/linux/signin.png
```
The platform is a segment in the baseline path. You have committed the first one. The second one does not exist, so on the runner every attempt 1 is a first run: it writes a baseline and fails, exactly as it did on your laptop the very first time. Attempt 2 then finds the file attempt 1 just wrote and passes.
The runner is thrown away at the end of the job, the written file goes with it, and tomorrow it happens again. "Flaky" is a fair description of what you are seeing and completely the wrong diagnosis.
answered by Nadia Belkacem
Why the platform is in the path at all: font rendering. Glyph rasterisation and subpixel antialiasing are not the same on macOS and on Linux, and text is most of the pixels on most pages. A baseline recorded on a laptop would not match a Linux runner even with byte-identical CSS and the same browser build — so rather than compare and fail, the path keeps them apart.
answered by Tomas BrekkeSDODS team
The fix is to produce the Linux baseline where CI produces it. Run the job once on a Linux runner with the update flag, then commit what it leaves behind:
bash
```
sdods run -p rwa-bank -e ci -l ui -b chromium -t @visual --update-snapshots
```
You commit __screenshots__/rwa-bank--ui--chromium/linux/ alongside the darwin one, and from then on both machines compare rather than write.
Two things I would do at the same time:
- Keep @visual out of the tag expression that gates a merge until the Linux baselines are in. A suite that goes flaky on a fresh runner teaches people to ignore red.
- Do not accept a baseline produced on a runner without opening it. It is a screenshot of your app taken by a machine nobody was watching.
This exact case is in Known limitations.
answered by Priya VenkateshSDODS team

Know the answer?

Wrap commands and output in triple backticks and they will render as a code block.

Checking whether you’re signed in…

Have a different question? Ask it.

@visual is green locally and reported flaky on CI — attempt 1 fails, attempt 2 passes

3 answers

Know the answer?

Related