335 of 390 scenarios failed at -w 4 with the user pool exhausted, and doctor is all green
Asked 9,840 views4 answers
52votes
Full regression at four workers. 55 passed, 335 failed, and nearly all of the failures are this:
text
SdodsError: All 1 user(s) with role "member" are leased.
hint: Owners: {"member":"nightly-20260401:2"}.
Increase env.users.poolSize, add users, or lower --workers.
Test timeout of 120000ms exceeded while running "beforeEach" hook.
The passes are all scenarios that need no user, or need @user:admin, which nothing else was competing for. About two thirds of our suite is @user:member.
We have one row per role in data/common/users.csv. I get that that is not many. What I do not get is why nothing said so — this is the whole check output:
doctor passes every check without ever comparing pool rows to worker count
Is one row per role simply not supported, or have I misconfigured the pool?
Your pool is configured correctly. It is too small, and the thing that makes it fatal rather than merely slow is that a lease is held per worker for the worker's whole lifetime, not per scenario. One row means exactly one worker can ever run a @user:member scenario. The other three wait out the lease timeout (30 s by default), throw, and because the throw happens inside beforeEach the test then runs on to the 120 s test timeout.
So each starved scenario costs you up to 150 seconds and produces no information. That is why your four-worker run was slower per test than a single-worker run would have been — fewer workers is currently both faster and more correct.
There is an open issue covering the framework half of this, and it is not fixed. Two things it asks for, so you know what is not going to save you today: the runner has every input it needs before the first test to refuse to start when workers exceeds the rows available for a role the selection uses, and it does not; and doctor does not compare rows-per-role against the worker count, which is why your screenshot is green.
What to do now, in order of preference:
Seed at least as many accounts per role as you intend to run workers. member carries most
of your suite, so start there. This is the real fix and it is on your side.
Until then pin -w 1. Put it in the invocation, not in someone's memory.
And re-baseline afterwards. A run contaminated like this tells you almost nothing about the product — every count you took from it is a count of the pool.
Before you take this to CI, there is a second and worse version of it there.
If your nightly runs a browser matrix as parallel jobs, each job is a separate machine with its own .sdods/leases directory. leaseStore: file is single-machine by design, and leaseStore: db is declared in the schema but has no implementation behind it — the file pool wins regardless. So three matrix jobs will lease the same real accounts at the same time, happily, with no contention visible to any of them.
If those accounts are shared identities on a real staging system, that is not just noisy results — two jobs mutating the same account's data will produce failures that are impossible to reproduce.
Until the shared lease store exists: run the matrix on one runner, or give each browser its own fixture identities, or accept single-browser nightlies. Also check your workflow actually pins --workers; with no flag it inherits the runner default, which on a standard hosted runner is enough to starve a one-row pool.
Data point since it sounds counter-intuitive until you see it: we pinned -w 1 on the nightly and the wall-clock time went down, from a bit over two hours to about ninety minutes. Four workers spent most of their lives asleep on a lease and then burning a test timeout.
One practical note on seeding: keep the pool rows in the dataset as ${TEST_*} references rather than literals, and add the new identities to whatever provisioning script created the first ones, with a verify mode. Otherwise you have four accounts that exist in a spreadsheet and one that exists in the system, and the next person to hit this has a harder problem than you did.
