Staging serving production’s robots.txt, and a sitemap missing the pricing page

Three separate problems, none of which a status-code check can see.

• The employee intranet path was absent from the disallow list while resolving (a 307 to auth, so not an exposure — but the URLs, and any login or error page under them, are crawlable and indexable). Two sibling paths were already disallowed, which makes this an omission rather than a decision.
• The primary acquisition page was missing from a seven-URL sitemap, while a lower-traffic page next to it was listed. Seven URLs is also low for a site with a blog, a templates gallery and a documentation site, so the generator is probably missing more than one page.
• Both files were byte-identical between staging and production. So the staging host named the production host in Host:, pointed Sitemap: at the production file, and listed production URLs — handing crawlers production canonical hints from a duplicate, publicly reachable copy of the marketing site. Nothing declared the staging host itself unindexable.

The third one is the interesting one for a test author, because the expected value differs per environment and the assertion the suite currently makes is the production expectation applied to staging.

These are text artefacts on the browser-facing origin, and you do not need a browser for them.

The API steps accept an absolute URL — anything matching ^https?:// is passed straight through instead of being joined to api.baseUrl — so an @api scenario can fetch a file from the UI host directly. Put that origin in envs/<env>.yaml under vars: and reference it through the template renderer:

vars:
  uiOrigin: https://${UI_BASE_URL}
  robotsDisallow: 'Disallow: /'

Then the response should contain {string} gives you a raw-body substring assertion, which is exactly the right granularity for a text file.

Two honest limits:

• there is no should not contain for a body or a header, so "staging must not advertise the production host" needs a project step. the response JSON path {string} should match {string} will not help — it is JSON only.
• @env:<name> is validated by lint against envs.available, so it documents the intent, but do not rely on it to keep a scenario off an environment at runtime. Select environments with -e and with a tag expression you control.

The design point is bigger than the assertion. Put the expected policy in vars: per environment and have one scenario read {{robotsDisallow}}. Then the environment-aware fix — serve Disallow: / and X-Robots-Tag: noindex on every non-production host — does not break the test that found the problem. Decide the contract before you write the assertion, or you will write it twice.

@api @smoke
Scenario: robots.txt states this environment's crawl policy
  Given I use no authentication
  When I send a GET request to "{{uiOrigin}}/robots.txt"
  Then the response status should be 200
  And the response should contain "{{robotsDisallow}}"
  And the response should contain "Disallow: /internal/"

@api @smoke
Scenario: The sitemap lists every acquisition surface
  Given I use no authentication
  When I send a GET request to "{{uiOrigin}}/sitemap.xml"
  Then the response status should be 200
  And the response should contain "{{uiOrigin}}/pricing"
  And the response should contain "{{uiOrigin}}/enterprise"

The second scenario asserts against {{uiOrigin}} rather than a literal host, which is what turns "the sitemap has the right pages" and "the sitemap describes the right environment" into one assertion instead of two.