← All questions

All 4 user(s) with role "standard" are leased — passes locally, four failures in CI

sdods run -p demo-shop -e staging -l ui -t @regression -w 4

USER_POOL_EXHAUSTED: All 4 user(s) with role "standard" are leased.

4 answers

• 36votes✓

  Accepted answer

  The unit is the worker, not the scenario.

  A lease is owned by runId:shardOffset+parallelIndex. The first scenario on a worker that needs a standard account takes one; the worker holds it for the rest of its life and releases at teardown. So the number of accounts with a role is a hard cap on how many workers can run that role's scenarios concurrently — four accounts, eight workers, four workers wait and then fail.

  That is the right model when scenarios mutate user-scoped state, which is the common case: two workers on one identity see each other's carts, sessions and sign-outs. Sharing accounts turns a passing suite into a flaky one and the failures look like application bugs.

  Three ways out, in the order I would try them:

  1. Seed more accounts with that role. The account count is a capacity number now — plan it like one.
  2. Cap -w at the number of accounts for your busiest role.
  3. Split the role's scenarios into their own run with its own worker count, and let the rest of the suite run wide.

  One thing to know before you go to CI with sharding: the owner key includes a shard offset so two shards never generate the same owner string. With leaseStore: file each machine has its own lease directory anyway, so three shards at four workers each need twelve accounts, not four. With leaseStore: db they contend for the same rows and the arithmetic is the same.

  PVanswered by Priya Venkateshmaintainer29,604 · 30 Jun 2022
• 12votes

  We hit this the week we turned on parallelism and the fix that stuck was making account count a CI concern rather than a repo concern: a pre-step calls the app's admin API and provisions N accounts per role, where N is the worker count, and the users CSV is generated from the response.

  It sounds heavier than it is, about forty lines, and it means nobody ever has to remember to add a row when they raise -w.

  MFanswered by Mina Farouk760 · 5 Jul 2022
• 7votes

  Careful with the obvious-looking fix: users.poolSize in the environment file does not create accounts. It caps how many the run may use. Raising it when the file has four rows changes nothing, and lowering it is a good way to reproduce this on purpose.

  NBanswered by Nadia Belkacem8,611 · 4 Aug 2022
• 4votes

  Something that saved us a lot of confusion — the wait is not instant failure, it is thirty seconds of retrying and then the error. So the symptom in CI is a suite that appears to hang near the end and then reports failures, which reads like a timeout problem rather than a capacity one. If your suite got slower before it got red, check the pool first.

  KPanswered by Kostya Petrov7,903 · 12 Sept 2022