Richie
50d4aa383f
treefmt / nix fmt (pull_request) Successful in 3s
test ebook search / test-ebook-search (pull_request) Failing after 8h50m21s
pytest / pytest (pull_request) Failing after 8h50m22s
build_systems / build-rhapsody-in-green (pull_request) Failing after 8h50m23s
build_systems / build-portal-1 (pull_request) Failing after 8h50m24s
build_systems / build-jeeves (pull_request) Failing after 8h50m24s
build_systems / build-brain (pull_request) Failing after 8h50m24s
build_systems / build-bob (pull_request) Failing after 8h50m24s
68 lines
2.9 KiB
Markdown
68 lines
2.9 KiB
Markdown
# pytest-xdist concurrent worker crashes
|
|
|
|
With two workers and a restart limit of three, the fourth worker crash
|
|
requests shutdown while another test can still be running. That test may
|
|
also crash. The original queued-work test requires exactly four failures,
|
|
even though five failures can occur without exceeding the replacement limit.
|
|
|
|
## Scope and behavior
|
|
|
|
`concurrent-worker-crashes.patch` changes the assertions in
|
|
`TestNodeFailure.test_max_worker_restart_tests_queued` in
|
|
`testing/acceptance_test.py`. It requires exactly three replacements, four or
|
|
five failed tests, the failed-tests exit status, the limit message, and no
|
|
internal error. It retains the two-worker workload and ten queued tests.
|
|
|
|
The existing nixpkgs pytest-9 compatibility patches remain in place.
|
|
Production scheduling and worker-restart behavior are unchanged.
|
|
|
|
## Reproduction and focused checks
|
|
|
|
Use a disposable pytest-xdist 3.8.0 checkout with its test dependencies and
|
|
the nixpkgs pytest-9 compatibility patches where required. From this directory:
|
|
|
|
```sh
|
|
patch --fuzz=0 -d /path/to/pytest-xdist -p1 < concurrent-worker-crashes.patch
|
|
cd /path/to/pytest-xdist
|
|
python -m pytest testing/acceptance_test.py \
|
|
-k test_max_worker_restart_tests_queued -q
|
|
```
|
|
|
|
Twenty unmodified runs passed during the review. To force the failing
|
|
schedule, modify the generated crashing test in a disposable checkout to
|
|
accept `worker_id`: make `gw3` wait for a marker created by `gw4`, and make
|
|
`gw4` pause 0.1 seconds after creating the marker. Then both have in-flight
|
|
tests when shutdown starts. Bound the marker wait so a reproduction failure
|
|
cannot hang the suite. The original assertion fails on five reported
|
|
failures; the patched test passes.
|
|
|
|
## Remaining resource settings
|
|
|
|
[`../../test-exclusions.nix`](../../test-exclusions.nix) runs the outer suite
|
|
with one worker and sets the inner-worker wait to 60 seconds. These settings
|
|
limit nested process pools and allow worker startup on loaded builders.
|
|
|
|
A separate reproduction inserts an 11-second `pytest_sessionstart` delay
|
|
into the child created by `test_basic_collect_and_runtests` in
|
|
`testing/test_remote.py`. The original 10-second channel wait fails; the
|
|
60-second wait passes. This is a worker-startup bound, not a product deadline.
|
|
|
|
## Upstream status
|
|
|
|
This is a standalone test patch for pytest-xdist 3.8.0. No upstream submission
|
|
was made during this work. Recheck the allowed in-flight failures and
|
|
replacement count when updating the scheduler or shutdown behavior.
|
|
|
|
## Local NixOS integration and build results
|
|
|
|
[`../default.nix`](../default.nix) loads `default.nix` through
|
|
`pythonPackagesExtensions`. From the repository root:
|
|
|
|
```sh
|
|
nix build --no-link -L .#nixosConfigurations.jeeves.pkgs.python314Packages.pytest-xdist
|
|
```
|
|
|
|
The patched x86-64-v3 package passed 185 tests, with 6 existing skips and
|
|
10 expected failures. The forced concurrent-crash reproduction passed after
|
|
the fix, and the focused test passed again after formatting the assertion.
|