perf.budgets validates and resolves, and a nine-second page still passes. What am I missing?

Example threads. These questions and answers were written by the SDODS team to document real problems and how to fix them. The people named are illustrative, not real users. Questions asked on this site are marked new.

Project yaml:

yaml

perf:
  budgets: { pageLoadMs: 4000, lcpMs: 3000, apiP95Ms: 1500 }

Environment override, and a process that asks for the gate:

yaml

processes:
  - name: release-gate
    trigger: release
    tags: '@smoke or @regression'
    gates: { minPassRate: 100, maxFlaky: 0, perfBudgets: true, a11y: true }

bash

sdods config show -p demo-shop -e staging --explain

shows perf.budgets.pageLoadMs resolving from envs/staging.yaml, so the plumbing is clearly there. But a page we know takes about nine seconds on staging goes green, and no scenario has ever failed on the gate. Is the budget compared against the wrong metric, or am I meant to add a step?

perf config ci

asked by Anouk de Vries

2 answers

Accepted answer
You are missing nothing, and the reason it looks like plumbing is that it is plumbing with nothing on the end of it.
The schema accepts perf.budgets. The schema accepts perfBudgets and a11y on a process gate. The resolver merges them and config show --explain will happily tell you which layer won. Then, as of this answer, nothing at run time reads either value. There is no timing check and no accessibility hook yet — the gate booleans are recorded and never asked.
So a green run is not evidence about the speed of that page. It is evidence that the scenario's assertions passed.
This is stated on the workshop chapter; the accessibility and performance guide contradicts it and the guide is the one that is wrong.
answered by Reeta NandalSDODS team
If the budget matters for a release, put it somewhere that enforces it — a separate step in the pipeline that measures and exits non-zero, owned by whoever cares about the number.
Keeping the yaml key costs nothing and documents the intent, so I would leave it. Just do not let anybody put "performance budgets enforced" on a slide because the key is in the file.
answered by Helle Borg

Know the answer?

Wrap commands and output in triple backticks and they will render as a code block.

Checking whether you’re signed in…

Have a different question? Ask it.

perf.budgets validates and resolves, and a nine-second page still passes. What am I missing?

2 answers

Know the answer?

Related