Thirty-Four Minutes to Two: What pytest-xdist Exposed in a Nearly 9,000-Test Suite

My agent fleet's test suite had grown to nearly 9,000 tests, and the CI Test step had grown with it: 34 minutes 40 seconds of a 35 minute 32 second job. Every pull request the fleet opened waited behind that number before a reviewer could even look at it.

The fix everyone reaches for is pytest-xdist, and it worked. The suite now runs in about two minutes without coverage and a little over three with it. But the speedup is the boring half of the story. The interesting half is what parallel execution exposed: three separate classes of shared state that had been hiding in a suite that was green every day, one of which only surfaced because a reviewer ran the parallel suite a few more times than the author did.

This is the account of that change, including the numbers, the worker count arithmetic, and the bugs.

The Setup: One Box, Three Runner Slots, and a Fleet

Some context matters for the decisions below. My fleet runs on a single dedicated machine with 48 logical cores. That box hosts a self-hosted GitHub Actions runner with up to three concurrent slots, and it also hosts the fleet's own task containers, where headless Claude Code agents pick up issues from a GitHub Projects board, write code, and open pull requests. A separate reviewer agent reads every pull request before it reaches me, and I merge from Telegram.

So CI is not a background chore here. It is the gate between an agent finishing a task and the next stage of the loop starting. A 35 minute job means a revise cycle (reviewer asks for changes, agent pushes a fix, CI reruns) costs over an hour of wall clock before anyone sees a result.

The measurements I am working from:

The serial no-coverage run passed 8,973 tests with 3 skipped. That count matters later.

Why 12 Workers and Not auto

The obvious first move with pytest-xdist is -n auto, which starts one worker per logical core. On this box that means 48 workers per CI run.

That would have been a mistake. The runner has three slots, so three pull requests in CI at the same time would start 144 pytest processes competing for 48 cores, and the fleet's own agent containers would be fighting them for the same CPU. A CI job that starves the agents producing the next pull request is not a win.

The executor that implemented this chose a fixed -n 12. In the worst case, three concurrent runs at 12 workers each is 36 processes, which stays inside the core count and leaves headroom for the fleet.

The more interesting justification came from the timing itself. A parallel run without coverage reported 15 minutes 58 seconds of user CPU time against 2 minutes 0 seconds of wall clock. That is roughly eight cores' worth of real work, even with twelve workers available. The tail of the run is dominated by tests that genuinely sleep, and no number of workers makes a real sleep shorter. Going higher than 12 would buy very little and cost contention.

The CI line itself is short:

run: >-
  pytest -q -n 12 --cov=agent_corp --cov-report=xml:coverage.xml
  ${{ github.event_name == 'push' && '--no-cov' || '' }}

Pull requests keep coverage, because a later step computes patch coverage from coverage.xml. Pushes to main skip it. The plugin version is pinned in the dev requirements file (pytest-xdist==3.8.0), and a test asserts that pin exists, the same way the linter version is pinned and guarded. Another test binds the shape of the workflow: a fixed worker count, coverage XML still produced on pull requests, and a separate small report-only step that must stay serial.

Shared State, Class One: Relative Paths in Settings

The first parallel run did not come back clean. One test, which checks that a revise cycle routes away from a degraded model to a fallback, failed intermittently under twelve workers and never serially.

The cause was a file called model_degradation.json. The daemon's settings object declares its state files with relative default paths, which resolve against the current working directory. In a test run that directory is the repository root. Under serial execution, every test that touched the file did so one at a time. Under xdist, all workers share the same working directory, so one worker wrote its degradation mark into the file while another worker's test was reading it back. The revise path loaded a neighbour's state over its own.

The tempting fix is to patch that one test. The executor instead treated the failure as a symptom of a class. A full serial run was already leaving ten stray JSON files in the checkout (pending reviews, pending decisions, a served-model baseline, and others), all from the same pattern. Thirty settings fields had relative defaults.

So the fix is an autouse fixture that walks the settings model and pins every relative path default into the test's own temporary directory:

@pytest.fixture(autouse=True)
def _isolate_relative_state_paths(monkeypatch, tmp_path):
    for name, field in Settings.model_fields.items():
        default = field.default
        if not isinstance(default, str) or not default:
            continue
        if not (name.endswith("_path") or name.endswith("_dir")
                or name.endswith("_file")):
            continue
        if os.path.isabs(default) or default == ".":
            continue
        monkeypatch.setattr(field, "default", str(tmp_path / default))
    Settings.model_rebuild(force=True)

A test that passes an explicit path still wins, because that happens after the fixture runs. Two tests that assert the settings defaults match module constants broke under the fixture (not under xdist), and they now compare against a snapshot of the pristine defaults captured at import time, before any fixture patches anything.

Shared State, Class Two: Files the Settings Never Knew About

The first revision went to the reviewer agent. Its finding was that the fixture fixed only the files the settings object knows about. Five more state files lived outside it and were still written to the shared repository root during a parallel run:

the daemon's restart markers and counters, resolved from an environment variable or the working directory at call time;

an LLM response cache in SQLite, resolved from an environment variable or the module's own location;

three JSONL logs for review metrics, review fan-out, and stats, defined as module constants relative to the source file.

The settings fixture structurally could not reach any of them. A second autouse fixture now pins the two environment variables (both read at call time, so setting the variable is enough) and the three module constants into the same per-test temporary directory. A guard test asserts that all seven resolved paths land inside tmp_path and never inside the checkout.

After this revision the suite was cleaner: no stray state files after any run and an empty git status each time. But in seven full parallel runs, one still failed. Same test as before, the degraded-model fallback. It did not reproduce in the next five runs. The executor recorded it as an open observation rather than retrying it away.

Shared State, Class Three: A Router That Never Got Put Back

The reviewer ran the parallel suite on its own and saw the same test fail twice in six runs. It then reduced the failure to a deterministic two-test order, and the cause turned out not to be a file at all.

The fleet's model router keeps its configuration in module globals: the ladder of model tiers, the CLI model name, and a few related values. Functions that configure the router rebind those globals rather than mutating an object a fixture could own. Most tests that reconfigure the router reset it afterwards. Several did not. One test, checking a cost-tracking edge case, narrowed the ladder to a single Sonnet tier and left it that way.

With no Sonnet tier on the ladder, the degraded-model fallback had nowhere to go. The router capped the choice back onto Opus, the degraded tier, and the revise launched on exactly the model the test expected it to avoid.

Serially, none of this showed. Alphabetical file order happened to put the tests that leave the router narrowed after the tests that read the default ladder. The suite had been passing on an ordering accident. Under xdist, load scheduling can put the polluter and the victim on the same worker in the wrong order, and it did, roughly one run in three.

Again, the fix targets the class, not the test. A third autouse fixture snapshots every router global at setup (copying containers, so in-place mutation hits the copy) and restores them at teardown through monkeypatch. It imposes no ladder of its own, so a test that configures the router explicitly still gets what it asked for.

Two guards keep it honest:

an AST test that reads the router module, collects every name declared global, and fails if the fixture's snapshot list misses one, so a future global cannot silently escape restoration;

a test that launches a child pytest process running the polluter and then the victim, which fails without the fixture and passes with it.

After this revision: seven consecutive green parallel runs, six plain and one with coverage, with identical counts every time.

Proving Nothing Fell Out of the Suite

A faster suite is worthless if it is faster because it runs fewer tests. Counting them is only half the question; the other half is whether each one asserts anything at all. The executor checked pass counts at every stage instead of trusting green:

serial on the merge base: 8,976 tests (8,973 passed, 3 skipped);

first parallel version: 8,980 on every run, which is 8,976 plus the 4 tests that version added;

final version: 8,983 on every run, which is 8,976 plus the 7 tests the change added in total.

Coverage on the pull request path still produced a readable coverage.xml (line rate 0.86), and the patch coverage step accepted it. No test was marked serial. No retry was added anywhere.

What It Looks Like a Week Later

The local numbers came from a task container on the same box. The executor could not read the Actions API from inside its container, so the runner-side "after" number had to come from CI itself. A week after the merge, on the self-hosted runner:

pushes to main (no coverage): Test step between 2m12s and 2m19s;

pull requests (with coverage): between 4m14s and 4m42s.

The pull request figure is higher than the local three and a bit minutes. Runs on the box overlap, and coverage tracing has a real cost. It is still roughly an eight times reduction on the path that gates every agent change, and the main branch path went from about half an hour to just over two minutes.

What I Take From It

Parallelism is a test of your test suite. Every failure that showed up under xdist was a real defect that already existed serially: files written into the checkout, state shared through the working directory, and module configuration leaking between tests. The serial suite was not correct; it was lucky.

Fix the class, then guard the class. Each of the three fixes is an autouse fixture that covers every instance of a pattern, plus a test that fails if a new instance slips past it. Patching the one failing test would have shipped a suite that flakes once a week.

Size workers to the machine you actually share. auto is a default for a machine that does nothing else. On a box that hosts the agents producing the work, a fixed worker count derived from slots times workers versus cores is a resource decision, and the CPU time numbers showed where the returns stopped.

An intermittent failure is a finding, not noise. The one-in-seven failure after the second revision was the trail to the router leak. If it had been retried away, the most interesting bug in the change would still be in the suite.

The independent review earned its keep. The author agent found the first class. The reviewer agent found the second by reading the change and the third by running the suite more often than the author had. I did not write any of this code. My part was reading the evidence and pressing merge.

FAQ

How much faster did the test step get with pytest-xdist?

Locally on the same machine, the suite went from about 29 minutes 50 seconds serial to about 2 minutes with twelve workers, and to about 3 minutes 20 seconds with coverage. On the self-hosted runner a week later, the main branch path takes a little over two minutes and pull requests take between four and five minutes, against 34 minutes 40 seconds before.

Why use a fixed worker count instead of auto?

The machine has 48 logical cores, up to three concurrent CI runs, and also hosts the agent containers. Auto would start 48 workers per run, up to 144 in total. Twelve workers per run caps the worst case at 36, and CPU time measurements showed the suite only used about eight cores of real work anyway, because the tail is dominated by tests that genuinely sleep. Sizing one shared machine instead of renting more of them is the same trade behind matching enterprise scale on a bootstrapped budget.

What broke when the suite ran in parallel?

Three kinds of shared state: relative file paths in the settings that resolved to the shared working directory, five more state files outside the settings, and router configuration in module globals that some tests changed and never restored. All three were already defects in the serial suite, hidden by sequential execution and alphabetical test order.

Were any tests marked serial or retried?

No. Every failure was fixed at its root with an autouse fixture, each fixture has a guard test, and pass counts were checked against the serial baseline at every revision to confirm nothing dropped out of the suite.

Related reading

Four Merge Presses, One Merge: Fixing the Approval Layer of My AI Agent Fleet

Two Runner Listeners, One Work Directory: A Self-Hosted CI Post-Mortem

Two Green Pull Requests, One Red Main: The Stale Merge Ref