Richie
7541387d7c
treefmt / nix fmt (pull_request) Successful in 4s
build_systems / prebuild-common-x86-64-v3 (pull_request) Successful in 24s
build_systems / build-portal-1 (pull_request) Successful in 20s
test ebook search / test-ebook-search (pull_request) Successful in 1m13s
build_systems / build-bob (pull_request) Successful in 54s
build_systems / build-brain (pull_request) Successful in 57s
build_systems / build-rhapsody-in-green (pull_request) Successful in 1m4s
build_systems / build-jeeves (pull_request) Successful in 1m30s
80 lines
3.5 KiB
Markdown
80 lines
3.5 KiB
Markdown
# pytest-xdist test fixes
|
|
|
|
With two workers and a restart limit of three, the fourth worker crash
|
|
requests shutdown while another test can still be running. That test may
|
|
also crash. The original queued-work test requires exactly four failures,
|
|
even though five failures can occur without exceeding the replacement limit.
|
|
|
|
## Scope and behavior
|
|
|
|
`concurrent-worker-crashes.patch` changes the assertions in
|
|
`TestNodeFailure.test_max_worker_restart_tests_queued` in
|
|
`testing/acceptance_test.py`. It requires exactly three replacements, four or
|
|
five failed tests, the failed-tests exit status, the limit message, and no
|
|
internal error. It retains the two-worker workload and ten queued tests.
|
|
|
|
`worker-startup-timeout.patch` changes the remote-test helper's event timeout
|
|
from 10 to 60 seconds so loaded builders have time to start workers. The
|
|
helper returns immediately when an event arrives and still has a bounded wait.
|
|
|
|
The existing nixpkgs pytest-9 compatibility patches remain in place.
|
|
Production scheduling and worker-restart behavior are unchanged.
|
|
|
|
## Reproduction and focused checks
|
|
|
|
Use a disposable pytest-xdist 3.8.0 checkout with its test dependencies and
|
|
the nixpkgs pytest-9 compatibility patches where required. From this directory:
|
|
|
|
```sh
|
|
patch --fuzz=0 -d /path/to/pytest-xdist -p1 < concurrent-worker-crashes.patch
|
|
patch --fuzz=0 -d /path/to/pytest-xdist -p1 < worker-startup-timeout.patch
|
|
cd /path/to/pytest-xdist
|
|
python -m pytest testing/acceptance_test.py \
|
|
-k test_max_worker_restart_tests_queued -q
|
|
python -m pytest testing/test_remote.py -q
|
|
```
|
|
|
|
Twenty unmodified runs passed during the review. To force the failing
|
|
schedule, modify the generated crashing test in a disposable checkout to
|
|
accept `worker_id`: make `gw3` wait for a marker created by `gw4`, and make
|
|
`gw4` pause 0.1 seconds after creating the marker. Then both have in-flight
|
|
tests when shutdown starts. Bound the marker wait so a reproduction failure
|
|
cannot hang the suite. The original assertion fails on five reported
|
|
failures; the patched test passes.
|
|
|
|
## Worker startup and outer concurrency
|
|
|
|
[`default.nix`](default.nix) runs the outer suite with one worker to limit
|
|
nested process pools. This is a Nix test-runner setting; the source timeout
|
|
change lives in [`worker-startup-timeout.patch`](worker-startup-timeout.patch).
|
|
|
|
A separate reproduction inserts an 11-second `pytest_sessionstart` delay
|
|
into the child created by `test_basic_collect_and_runtests` in
|
|
`testing/test_remote.py`. The original 10-second channel wait fails; the
|
|
60-second wait passes. This bounds waits for test worker events, including
|
|
startup, rather than changing a product deadline.
|
|
|
|
## Upstream status
|
|
|
|
These are standalone test patches for pytest-xdist 3.8.0. No upstream submission
|
|
was made during this work. Recheck the allowed in-flight failures, replacement
|
|
count, and remote-test wait when updating the scheduler or worker behavior.
|
|
|
|
## Local NixOS integration and build results
|
|
|
|
[`../default.nix`](../default.nix) loads `default.nix` through
|
|
`pythonPackagesExtensions`. From the repository root:
|
|
|
|
```sh
|
|
nix build --no-link -L .#nixosConfigurations.jeeves.pkgs.python314Packages.pytest-xdist
|
|
```
|
|
|
|
After consolidating the settings in this directory, the full x86-64-v3 package
|
|
build passed 185 tests, with 6 existing skips and 10 expected failures. Nix
|
|
evaluation confirmed the same outer-worker limit and preserved existing
|
|
patches, with the timeout now applied as a source patch.
|
|
|
|
The earlier forced concurrent-crash and delayed-startup reproductions passed
|
|
after their fixes; the focused crash test also passed after formatting its
|
|
assertion.
|