Killed a run halfway through. The next run fails immediately-ish with:
USER_POOL_EXHAUSTED: All 4 user(s) with role "standard" are leased.Nothing is running. .sdods/leases/demo-shop/staging/ has four .lock files in it, each with an owner and a timestamp from the run I killed.
Am I supposed to delete these by hand? It feels like the kind of thing that should not need a human, but I also do not want to delete a lock some other person's run is holding.
Accepted answer
You can delete them and it is safe when nothing is running, but you do not have to — the wait it out path works.
Each lock holds the owner and the time it was taken. When a worker wants an account and finds the lock held by somebody else, it compares the age against data.userPool.leaseTtlMs and, if the lease is older than that, treats it as abandoned, removes it, and takes the account. Default is ten minutes, so a cancelled run costs you at most ten minutes of confusion.
Deleting the directory is the fast path and what I do locally.
leaseTtlMs to make recovery quicker. The TTL is also the answer to "how long may a live worker legitimately hold this", so a TTL shorter than your longest scenario lets a second worker steal an account out from under a running test — and that failure looks like an application bug, not a configuration one.For CI it is a non-problem: ephemeral runners start with an empty workspace, so there are no locks to inherit. We still clear the directory in the pre-step for the self-hosted pool, one line, and it has never cost us anything.
This is really a local-machine problem, and only with leaseStore: file.
Following on: do not reach for leaseStore: db to make leases visible across machines. It is in the schema, but as of today nothing implements it and the file store runs whatever you set. If you are sharding across machines against a shared account estate, the file store is not doing what you think it is doing.
Have a different question? Ask it.