295 API scenarios fail with QUOTA_EXCEEDED: Exceeded quota for verifying passwords

Example threads. These questions and answers were written by the SDODS team to document real problems and how to fix them. The people named are illustrative, not real users. Questions asked on this site are marked new.

Ran the API and hybrid layers, single worker, one browser. 1,152 tests. 295 of the failures are this one message:

text

Error: Firebase sign-in failed for qa-member@example.com:
  QUOTA_EXCEEDED : Exceeded quota for verifying passwords.

The credentials are fine. If I run any one of those scenarios on its own an hour later it passes. The failures also start partway through the run rather than at the beginning, which is what made me think it was a rate limit rather than a password.

Our project has a step that mints a session by calling the identity provider directly, because an @api scenario cannot use the browser storage state. Is there a supported way to stop it doing that once per scenario?

api pool ci

asked by Ade Oyinlola

4 answers

Accepted answer
Not one that ships working today, no. You are going to have to cache it yourself, and the good news is that it is a small change with a very large effect.
The shape of the problem: that step performs a fresh password sign-in for every scenario. Identity Platform rate-limits password verification per project, so with several hundred @api scenarios you are making several hundred verifications against one project from one IP within a few minutes, and you exhaust the quota partway through. Everything after that fails at sign-in, before any assertion runs. Because it is time-dependent it looks exactly like flake — a scenario that fails at minute twenty passes alone — which is how this gets closed as "flaky" and comes back.
An ID token from that endpoint is valid for an hour. Cache one per role, keyed the way the UI auth cache is, with a TTL comfortably under the hour — 45 minutes matches the auth.maxAgeMinutes the UI side already uses for the same reason. That turns several hundred sign-ins into one per role.
Until you have it, keep API runs single-worker and single-browser so you are at least not multiplying the rate, and be aware that a run in this state tells you almost nothing about your API surface. The scenarios that passed are the ones that ran before the quota went.
answered by Priya VenkateshSDODS team
Worth knowing that the mechanism which should make Priya's answer unnecessary already ships, and does nothing.
sdods auth capture writes a token file per role and index:
text
```
projects/<slug>/.auth/staging/owner-0.token.json
projects/<slug>/.auth/staging/admin-0.token.json
projects/<slug>/.auth/staging/member-0.token.json
```
Those files are on your disk right now and they contain what the API layer needs. Nothing reads them at runtime, and auth.tokenPlacement has no consumer either. That is the open issue — the capture side is written, the consumption side is not — and it is why every project that hits this ends up hand-rolling the cache instead of using the framework's own path.
So: build the cache, but build it so you can delete it later.
answered by Daniel OyelaranSDODS team
Please also check what else lives on that identity project before you keep running into the limit. Quota exhaustion is per project, not per caller — if anything real signs in against the same project as your staging fixtures, your test suite can lock out actual people for the rest of the window. We moved the fixtures onto their own project partly for that reason and partly so the noise in the sign-in metrics stopped being ours.
answered by Rasha Halabi
Small addition to the cache: handle the failure as well as the success. Backoff-and-retry specifically on QUOTA_EXCEEDED turns a cliff into a slope, so if the cache misses in a burst (new role, expired TTL, whatever) you degrade instead of losing the rest of the run. Ten lines, and it saved us a rerun the first week it existed.
answered by Sunita Kale

Know the answer?

Wrap commands and output in triple backticks and they will render as a code block.

Checking whether you’re signed in…

Have a different question? Ask it.

295 API scenarios fail with QUOTA_EXCEEDED: Exceeded quota for verifying passwords

4 answers

Know the answer?

Related