All 4 user(s) with role "standard" are leased — passes locally, four failures in CI

Example threads. These questions and answers were written by the SDODS team to document real problems and how to fix them. The people named are illustrative, not real users. Questions asked on this site are marked new.

Suite is green on my machine and fails in CI, and the difference is the worker count. Local:

bash

sdods run -p demo-shop -e staging -l ui -t @regression -w 4

CI runs the same thing with -w 8. Four scenarios pass, four sit there for what looks like half a minute and then fail with:

text

USER_POOL_EXHAUSTED: All 4 user(s) with role "standard" are leased.

There are four standard accounts in data/staging/users.csv. I assumed a lease was per scenario, so eight workers would just take turns — a scenario takes six seconds, there is plenty of time. It clearly does not work like that. What is the actual unit?

pool data ci

asked by Owen Brackley

4 answers

Accepted answer
The unit is the worker, not the scenario.
A lease is owned by runId:shardOffset+parallelIndex. The first scenario on a worker that needs a standard account takes one; the worker holds it for the rest of its life and releases at teardown. So the number of accounts with a role is a hard cap on how many workers can run that role's scenarios concurrently — four accounts, eight workers, four workers wait and then fail.
That is the right model when scenarios mutate user-scoped state, which is the common case: two workers on one identity see each other's carts, sessions and sign-outs. Sharing accounts turns a passing suite into a flaky one and the failures look like application bugs.
Three ways out, in the order I would try them:
1. Seed more accounts with that role. The account count is a capacity number now — plan it like one.
2. Cap -w at the number of accounts for your busiest role.
3. Split the role's scenarios into their own run with its own worker count, and let the rest of the suite run wide.
One thing to know before you go to CI with sharding: the owner key includes a shard offset so two shards never generate the same owner string. With leaseStore: file each machine has its own lease directory anyway, so three shards at four workers each need twelve accounts, not four. leaseStore: db is in the schema too, but nothing implements it yet, so do the arithmetic per machine.
answered by Priya VenkateshSDODS team
We hit this the week we turned on parallelism and the fix that stuck was making account count a CI concern rather than a repo concern: a pre-step calls the app's admin API and provisions N accounts per role, where N is the worker count, and the users CSV is generated from the response.
It sounds heavier than it is, about forty lines, and it means nobody ever has to remember to add a row when they raise -w.
answered by Mina Farouk
Careful with the obvious-looking fix: users.poolSize in the environment file does not create accounts. It caps how many the run may use. Raising it when the file has four rows changes nothing, and lowering it is a good way to reproduce this on purpose.
answered by Nadia Belkacem
Something that saved us a lot of confusion — the wait is not instant failure, it is thirty seconds of retrying and then the error. So the symptom in CI is a suite that appears to hang near the end and then reports failures, which reads like a timeout problem rather than a capacity one. If your suite got slower before it got red, check the pool first.
answered by Kostya Petrov

Know the answer?

Wrap commands and output in triple backticks and they will render as a code block.

Checking whether you’re signed in…

Have a different question? Ask it.

All 4 user(s) with role "standard" are leased — passes locally, four failures in CI

4 answers

Know the answer?

Related