Added @a11y to two existing scenarios expecting them to get slower and probably to fail, because the pages in question are not good.
sdods run -p demo-shop -e staging -l ui -b chromium -t @a11yBoth passed, in about the same time as before. I cannot tell from the output whether an accessibility pass ran and found nothing, or nothing ran. Is there something in the report that would distinguish the two?
Accepted answer
Yes, and it will tell you nothing ran.
An accessibility pass would attach JSON per step under sdods/a11y/<step>. Open the run, go to Artifacts, and there is nothing under that name. Empty artefacts is the difference between "checked and clean" and "not checked", and it is the only reliable tell.
@a11y is a plain selection tag. In this area the only tag the runner treats specially is @visual, which chooses the screenshot policy; everything else a scenario carries is used to decide whether it runs, not what happens while it does. Your two scenarios ran as ordinary scenarios and passed for the ordinary reason.
And this is why the empty-artefact tell matters more than it sounds. A green accessibility test that runs nothing is worse than having no accessibility test, because it is a claim — on a dashboard, in a report to somebody who cannot check it, eventually in a procurement answer.
Rename the tag, add a comment above the scenario, do whatever your team will actually read. Just do not leave @a11y sitting there looking like coverage.
The thing that has genuinely caught accessibility regressions for us, in the absence of a checker, is the locator strategy.
Scenarios written against role and accessible name break the moment the accessible name disappears — somebody swaps a labelled button for a div with a click handler and four scenarios go red for what looks like a locator reason and is actually the finding.
It is not a substitute. It says nothing about contrast, focus order, or heading structure. But it is not nothing, and it is the part you already have.
Have the same question?
Ask it with the command you ran and the output you got, and it will be answered here.
Ask a question