Setting up a soak job: run the regression five times over, spread across four machines, and fail if anything is flaky.
sdods run -p demo-shop -e staging -t @regression --repeat-each 5 --shard 1/4 --run-id soak-0811 --reporter blob --fail-on-flakyWhat I cannot work out is whether the repeats are part of the set that gets sharded — so each machine gets a quarter of 5N tests — or whether each shard takes a quarter of the N scenarios and then repeats each of them five times locally. The wall-clock maths is the same either way, but the flakiness answer is not: repeats spread across four machines exercise four different environments, and repeats on one machine exercise the same one four times, which is a weaker test of exactly what I am trying to measure.
Has anyone actually watched this happen rather than reasoned about it? I can instrument it, but if someone already knows I would rather not.