Skip to content

← All use cases

A coverage manifest, a CI guard, and the gap between planned and delivered

A green coverage guard over nine feature files that do not exist

What makes it hard

The guard ran in CI, printed the drift in its own output, and exited zero:

text
coverage: 87/87 modules implemented · 269 feature file(s) · 1608 scenario(s) on disk · 1640 planned
✅ every feature file belongs to a declared module.

It asserted one direction — every file on disk belongs to a declared module — and never the reverse. Nine planned feature files, thirty-eight scenarios, were never written. The numbers needed to fail were already computed and on the screen.

Two of the nine mattered far more than the rest.

The suite never completed a purchase. Thirteen checkout scenarios existed and every one stopped at the session URL. Nothing proved that a payment completes, that a subscription row is written, that entitlements change, or that the upgrade prompt goes away. For a product whose revenue path is a hosted checkout, that is the single most important scenario in the repository and it was planned, counted and never built.

The deny direction of a catalogue was dropped. "What must NOT be offered" was a planned file; what shipped was a generated positive-direction outline over what is offered. Those are not substitutes, and the deny half is the half that catches a gate failing open.

Three files on disk were absent from the manifest, consistent with a consolidation nobody recorded — which is the deeper problem. A planned file that was deliberately merged and a planned file nobody wrote look identical in a directory listing.

How to cover it

sdods coverage measures a plan, which is the thing a file-count guard structurally cannot do.

bash
sdods coverage -p <slug> -e staging --routes --endpoints --roles --uncovered

It joins three declared things — the routes in sdods.project.yaml and its modules, each module's endpoints: (and, with --openapi, the paths of the spec named by env.api.openapi) and the pool roles: — against the scenarios that reference them, split by suite tag. --uncovered names what nothing touches, and it exits 1 when a module has no covered target.

The sdods coverage table, showing declared routes and endpoints against the scenarios that reference them
sdods coverage joins the plan to the scenarios, not the files to the modules

So declare the target first. A route that exists in the module and in no scenario becomes a red build, not a line of output nobody reads. That inverts the guard's direction for free, and it does it against the product's own surface rather than against a hand-written manifest that drifts.

Around that:

  • record a consolidation as a consolidation. A per-file status — implemented, consolidated into another path, or dropped with a reason — is what separates a decision from an omission.
  • make the guard fail on the drift it already prints. It computed 269 against 1,640; the only missing part was an exit code.
  • keep the deny direction as its own file. If it is folded into a generated positive outline it will not come back.

One honest framework note on the checkout gap: there is no iframe step in the shared library, so a card field rendered inside a payment iframe is not reachable from Gherkin today. BasePage exposes frame(name), so a project step in a page object can reach it — but the shared library will not do it for you, and pretending otherwise is how a planned file stays planned.

Scenario sketch

bash
sdods coverage -p <slug> -e staging --uncovered
sdods features list -p <slug> --scenarios
gherkin
@hybrid @regression
Scenario: A completed purchase writes the subscription and lifts the gate
  Given I use a leased user with role "purchaser"
  When I seed via POST "/billing/checkout" with body:
    """json
    { "plan": "starter", "workspaceId": "{{purchaserWorkspaceId}}" }
    """
  And I save the response JSON path "sessionUrl" as "checkoutUrl"
  And I complete the hosted checkout with the test card
  When I poll GET "/billing?workspaceId={{purchaserWorkspaceId}}" until JSON path "plan" equals "starter" within 60 seconds
  Then the response JSON path "isPaid" should equal "true"
  When I navigate to the "workspace" page
  Then I should not see the text "Upgrade to create more"

I complete the hosted checkout with the test card is a project step wrapping BasePage.frame(). Everything else on that scenario is the shared library, including the poll — which is the right way to wait for a webhook to land, and the reason this reads as @hybrid rather than as two disconnected tests.

Other use cases