@visual is green locally and reported flaky on CI — attempt 1 fails, attempt 2 passes

Asked 19 May 202221,400 views3 answers

58votes

Same commit, same command, two machines.

On my laptop:

bash

sdods run -p rwa-bank -e local -l ui -b chromium -t @visual

Green, every time. On the CI job (Ubuntu runner, project has retries: { ci: 2 }) the scenario is reported flaky. Attempt 1:

text

Error: A snapshot doesn't exist at projects/rwa-bank/features/__screenshots__/rwa-bank--ui--chromium/linux/signin.png, writing actual.

Attempt 2 passes and the run finishes green-with-a-flaky. Nothing in git changed between the attempts, obviously. This has happened on every nightly for a week and I do not understand what is unstable.

visual ci linux

asked by Sam Orbison9,420 · 19 May 2022

3 answers

71votes✓
Accepted answer
Nothing is unstable. Read the two paths next to each other:
text
```
projects/rwa-bank/features/__screenshots__/rwa-bank--ui--chromium/darwin/signin.png
projects/rwa-bank/features/__screenshots__/rwa-bank--ui--chromium/linux/signin.png
```
The platform is a segment in the baseline path. You have committed the first one. The second one does not exist, so on the runner every attempt 1 is a first run: it writes a baseline and fails, exactly as it did on your laptop the very first time. Attempt 2 then finds the file attempt 1 just wrote and passes.
The runner is thrown away at the end of the job, the written file goes with it, and tomorrow it happens again. "Flaky" is a fair description of what you are seeing and completely the wrong diagnosis.
answered by Nadia Belkacem8,611 · 19 May 2022
26votes
Why the platform is in the path at all: font rendering. Glyph rasterisation and subpixel antialiasing are not the same on macOS and on Linux, and text is most of the pixels on most pages. A baseline recorded on a laptop would not match a Linux runner even with byte-identical CSS and the same browser build — so rather than compare and fail, the path keeps them apart.
answered by Tomas Brekkemaintainer33,915 · 20 May 2022
34votes
The fix is to produce the Linux baseline where CI produces it. Run the job once on a Linux runner with the update flag, then commit what it leaves behind:
bash
```
sdods run -p rwa-bank -e ci -l ui -b chromium -t @visual --update-snapshots
```
You commit __screenshots__/rwa-bank--ui--chromium/linux/ alongside the darwin one, and from then on both machines compare rather than write.
Two things I would do at the same time:
- Keep @visual out of the tag expression that gates a merge until the Linux baselines are in. A suite that goes flaky on a fresh runner teaches people to ignore red.
- Do not accept a baseline produced on a runner without opening it. It is a screenshot of your app taken by a machine nobody was watching.
This exact case is in Known limitations.
answered by Priya Venkateshmaintainer29,604 · 2 Jun 2022

Have the same question?

Ask it with the command you ran and the output you got, and it will be answered here.

Ask a question

@visual is green locally and reported flaky on CI — attempt 1 fails, attempt 2 passes

3 answers

Related