moved rhapsody-in-green off v3
treefmt / nix fmt (pull_request) Successful in 4s
build_systems / prebuild-common-x86-64-v3 (pull_request) Successful in 24s
build_systems / build-portal-1 (pull_request) Successful in 20s
test ebook search / test-ebook-search (pull_request) Successful in 1m13s
build_systems / build-bob (pull_request) Successful in 54s
build_systems / build-brain (pull_request) Successful in 57s
build_systems / build-rhapsody-in-green (pull_request) Successful in 1m4s
build_systems / build-jeeves (pull_request) Successful in 1m30s
treefmt / nix fmt (pull_request) Successful in 4s
build_systems / prebuild-common-x86-64-v3 (pull_request) Successful in 24s
build_systems / build-portal-1 (pull_request) Successful in 20s
test ebook search / test-ebook-search (pull_request) Successful in 1m13s
build_systems / build-bob (pull_request) Successful in 54s
build_systems / build-brain (pull_request) Successful in 57s
build_systems / build-rhapsody-in-green (pull_request) Successful in 1m4s
build_systems / build-jeeves (pull_request) Successful in 1m30s
This commit is contained in:
@@ -1,9 +0,0 @@
|
||||
{
|
||||
nixpkgs.hostPlatform = {
|
||||
system = "x86_64-linux";
|
||||
gcc = {
|
||||
arch = "x86-64-v3";
|
||||
tune = "generic";
|
||||
};
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
nixpkgs = {
|
||||
hostPlatform = {
|
||||
system = "x86_64-linux";
|
||||
gcc = {
|
||||
arch = "x86-64-v3";
|
||||
tune = "generic";
|
||||
};
|
||||
};
|
||||
|
||||
# These patches repair tests and bundled dependencies that are sensitive
|
||||
# to the compiler flags used by the x86-64-v3 package set.
|
||||
overlays = [ (import ./patches) ];
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
# Package patches
|
||||
|
||||
Each package follows the [GnuTLS layout](gnutls/README.md):
|
||||
|
||||
- `default.nix` applies the patch through the package overlay.
|
||||
- A descriptive `.patch` file contains the standalone upstream source change.
|
||||
- `README.md` explains the problem, scope, reproduction, upstream status,
|
||||
Nix integration, and recorded validation limits.
|
||||
- Companion `verify-*` tools live beside the patch when needed; otherwise
|
||||
the README gives commands for the package's existing tests.
|
||||
|
||||
Keep package-specific evidence in its directory. Patch headers explain the
|
||||
change independently of Nix, and `default.nix` preserves existing patches.
|
||||
|
||||
| Package | Repair |
|
||||
| --- | --- |
|
||||
| [Abseil](abseil/README.md) | Public BMI2 header in Electron, Deno, and Signal's vendored copies |
|
||||
| [Backrefs](backrefs/README.md) | Match the regex timeout's CPU clock |
|
||||
| [GnuTLS](gnutls/README.md) | Wait for the UDP server socket before connecting |
|
||||
| [Jupyter Server](jupyter-server/README.md) | Exercise the correct shared future during reconnect |
|
||||
| [Prometheus](prometheus/README.md) | Complete parsing before inspecting the test editor state |
|
||||
| [pytest-xdist](pytest-xdist/README.md) | Check worker replacements and allow startup on loaded builders |
|
||||
| [SciPy](scipy/README.md) | Account for floating-point rounding in STFT tests |
|
||||
| [Sentry SDK](sentry-sdk/README.md) | Isolate SDK thread mocks from Python's threading module |
|
||||
| [Torchaudio](torchaudio/README.md) | Compare pitch-shift batches at appropriate precision |
|
||||
| [TorchCodec](torchcodec/README.md) | Match the reference MP3 encoder's sample format |
|
||||
|
||||
## Local NixOS integration
|
||||
|
||||
The [`x86-64-v3` optional module](../../common/optional/x86-64-v3/default.nix)
|
||||
imports this directory's [`default.nix`](default.nix) directly, so the patch
|
||||
overlay applies only to hosts using that package set. Prometheus patches its
|
||||
separate assets derivation; Python packages use `pythonPackagesExtensions`.
|
||||
|
||||
The [pytest-xdist directory](pytest-xdist/README.md) also owns its outer-worker
|
||||
limit and remote-worker event timeout. These package overrides add no skipped
|
||||
tests. Existing nixpkgs exclusions remain separate from these repairs.
|
||||
|
||||
The test-exclusion review used Python 3.14.7 and the pinned x86-64-v3 package
|
||||
set. Host-flake evaluation verified patch wiring, Python install checks,
|
||||
removal of the local skips, and Prometheus's reference to the patched assets.
|
||||
Jupyter and Sentry package tests used the preceding dependency set with the
|
||||
new package patch to avoid unrelated rebuilds after pytest-xdist changed.
|
||||
No complete NixOS rebuild was performed. Individual READMEs distinguish
|
||||
package builds, focused tests, and checks that have not been run.
|
||||
@@ -0,0 +1,59 @@
|
||||
# Abseil BMI2 public header
|
||||
|
||||
Vendored Abseil includes `bmi2intrin.h` directly when `__BMI2__` is enabled.
|
||||
Compilers reject that internal header without the umbrella-header setup.
|
||||
`bmi2-public-header.patch` includes `immintrin.h` instead, allowing builds
|
||||
that enable BMI2 through `-march=x86-64-v3`.
|
||||
|
||||
## Scope and behavior
|
||||
|
||||
The patch changes one include in
|
||||
`third_party/abseil-cpp/absl/container/internal/raw_hash_set.h`.
|
||||
`default.nix` applies it to Electron 43's unwrapped package, Deno's
|
||||
`librusty_v8`, and Signal's WebRTC dependency. It also supplies the patched
|
||||
Electron package to Signal. These overrides apply only to `x86-64-v3`.
|
||||
|
||||
The shared file path is relative to each vendoring project's source root,
|
||||
not the root of a standalone Abseil checkout. No hash-table algorithm or
|
||||
test exclusion changes.
|
||||
|
||||
## Reproduction and focused checks
|
||||
|
||||
From this directory, check and apply the patch to each vendored source tree:
|
||||
|
||||
```sh
|
||||
patch --dry-run --fuzz=0 -d /path/to/vendor-source -p1 < bmi2-public-header.patch
|
||||
patch --fuzz=0 -d /path/to/vendor-source -p1 < bmi2-public-header.patch
|
||||
```
|
||||
|
||||
A small compiler check isolates the header requirement. With GCC or Clang
|
||||
on x86-64, compile `#include <bmi2intrin.h>` using `-march=x86-64-v3`; the
|
||||
compiler rejects the direct include. Changing it to `#include <immintrin.h>`
|
||||
should compile. The full consumer builds below check integration with their
|
||||
actual toolchains.
|
||||
|
||||
## Upstream status
|
||||
|
||||
Abseil addressed this issue through
|
||||
[PR #2071](https://github.com/abseil/abseil-cpp/pull/2071), imported by its
|
||||
upstream workflow. That change uses `x86gprintrin.h`; this local variant uses
|
||||
the public `immintrin.h` umbrella header for the vendored toolchains.
|
||||
Keep the workaround until all three bundled copies include a compatible fix.
|
||||
This file is a local adaptation, not a verbatim copy of the upstream diff.
|
||||
|
||||
## Local NixOS integration and build results
|
||||
|
||||
[`../default.nix`](../default.nix) merges this directory's overlay fragment
|
||||
because it repairs multiple packages. From the repository root, the consumer
|
||||
build commands are:
|
||||
|
||||
```sh
|
||||
nix build --no-link -L .#nixosConfigurations.jeeves.pkgs.deno
|
||||
nix build --no-link -L .#nixosConfigurations.jeeves.pkgs.electron_43
|
||||
nix build --no-link -L .#nixosConfigurations.jeeves.pkgs.signal-desktop
|
||||
```
|
||||
|
||||
The earlier extraction checked the vendored header snapshots and evaluated
|
||||
all three patch attachments. Those records do not establish successful full
|
||||
consumer rebuilds. No new compiler or consumer build was run for the layout
|
||||
change; the patch and override are unchanged.
|
||||
@@ -0,0 +1,20 @@
|
||||
Subject: [PATCH] abseil: include BMI2 intrinsics through the public header
|
||||
|
||||
GCC and Clang reject direct inclusion of bmi2intrin.h. Include immintrin.h
|
||||
instead so that the compiler supplies the required intrinsic setup when
|
||||
BMI2 is enabled, including builds targeting x86-64-v3.
|
||||
|
||||
This patch is shared by the vendored Abseil copies in Electron, rusty_v8
|
||||
(Deno), and Signal's WebRTC build.
|
||||
|
||||
--- a/third_party/abseil-cpp/absl/container/internal/raw_hash_set.h
|
||||
+++ b/third_party/abseil-cpp/absl/container/internal/raw_hash_set.h
|
||||
@@ -226,7 +226,7 @@
|
||||
#endif
|
||||
|
||||
#ifdef __BMI2__
|
||||
-#include <bmi2intrin.h>
|
||||
+#include <immintrin.h>
|
||||
#endif // __BMI2__
|
||||
|
||||
namespace absl {
|
||||
@@ -0,0 +1,38 @@
|
||||
# Abseil accepted the upstream fix: https://github.com/abseil/abseil-cpp/pull/2071
|
||||
# Keep this workaround until Electron, Deno's rusty_v8, and Signal's WebRTC
|
||||
# update their bundled Abseil copies to include it.
|
||||
{ prev }:
|
||||
let
|
||||
patchAbseilBmi2Include =
|
||||
package:
|
||||
package.overrideAttrs (old: {
|
||||
# GCC and Clang require the public umbrella header for BMI2 intrinsics.
|
||||
patches = (old.patches or [ ]) ++ [ ./bmi2-public-header.patch ];
|
||||
});
|
||||
|
||||
electron43Unwrapped = patchAbseilBmi2Include prev.electron_43.unwrapped;
|
||||
electron43 = prev.electron_43.override {
|
||||
electron-unwrapped = electron43Unwrapped;
|
||||
};
|
||||
|
||||
signalCallPackage =
|
||||
path: args:
|
||||
let
|
||||
package = prev.callPackage path args;
|
||||
in
|
||||
if builtins.baseNameOf path == "webrtc.nix" then patchAbseilBmi2Include package else package;
|
||||
in
|
||||
prev.lib.optionalAttrs ((prev.stdenv.hostPlatform.gcc.arch or null) == "x86-64-v3") {
|
||||
deno =
|
||||
let
|
||||
librusty_v8 = patchAbseilBmi2Include prev.deno.passthru.librusty_v8;
|
||||
in
|
||||
prev.deno.override { inherit librusty_v8; };
|
||||
|
||||
electron_43 = electron43;
|
||||
|
||||
signal-desktop = prev.signal-desktop.override {
|
||||
electron_43 = electron43;
|
||||
callPackage = signalCallPackage;
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
_final: prev:
|
||||
(import ./abseil { inherit prev; })
|
||||
// {
|
||||
gnutls = import ./gnutls { inherit (prev) gnutls; };
|
||||
prometheus = import ./prometheus { inherit (prev) prometheus; };
|
||||
|
||||
pythonPackagesExtensions = prev.pythonPackagesExtensions ++ [
|
||||
(_pythonFinal: pythonPrev: {
|
||||
pytest-xdist = import ./pytest-xdist { inherit (pythonPrev) pytest-xdist; };
|
||||
scipy = import ./scipy { inherit (pythonPrev) scipy; };
|
||||
sentry-sdk = import ./sentry-sdk { inherit (pythonPrev) sentry-sdk; };
|
||||
})
|
||||
];
|
||||
}
|
||||
@@ -0,0 +1,160 @@
|
||||
# GnuTLS UDP server readiness
|
||||
|
||||
Under load, the test client can start before `gnutls-serv` binds its UDP
|
||||
socket, and the first handshake fails with `Connection refused`.
|
||||
`serv-udp.sh` currently waits a fixed four seconds; elapsed time does not
|
||||
establish server readiness. `udp-server-readiness.patch` replaces that wait
|
||||
with polling for the local IPv4 UDP endpoint.
|
||||
|
||||
## Scope and waiting behavior
|
||||
|
||||
The patch changes the existing `wait_udp_server()` and adds a new
|
||||
`check_if_udp_port_bound()` beside it in `tests/scripts/common.sh`.
|
||||
`serv-udp.sh` is its only caller in 3.8.13. The TCP helpers `wait_server()`
|
||||
and `wait_for_port()`, including their existing sleeps, are unchanged.
|
||||
Both original DTLS handshake checks remain unchanged.
|
||||
|
||||
Each iteration checks process liveness and the socket **before sleeping**.
|
||||
A ready socket returns immediately. An unsuccessful check sleeps two
|
||||
seconds only if another attempt remains: at most 90 attempts, consistent
|
||||
with the existing `wait_server()` budget implemented by `wait_for_port()`,
|
||||
with no sleep after the final check. Server exit fails early; exhausting the
|
||||
budget fails and terminates the server. No handshake is retried, and no
|
||||
protocol timeout is changed. Once bound, the kernel can queue datagrams
|
||||
while the server is scheduled; the probe itself sends no packets.
|
||||
|
||||
The existing `have_port_finder()` prefers `ss`, then `netstat`. If neither
|
||||
exists, it prints `neither ss nor netstat found` and exits **77 (skip)**.
|
||||
In the normal test flow, port selection calls it before launching a server.
|
||||
The probe runs in a subshell so that, even if this skip occurs after launch,
|
||||
the waiting helper can terminate and reap the server before exiting 77.
|
||||
|
||||
## Why an IPv4 socket is expected
|
||||
|
||||
This is specific to the server used by this test, not a general rule that
|
||||
IPv6 sockets cannot serve IPv4 clients. The client explicitly uses
|
||||
`127.0.0.1`. The server's `--udp` path calls `udp_server()`, which calls
|
||||
`listen_socket(..., SOCK_DGRAM)`. That function iterates the wildcard
|
||||
addresses returned by `getaddrinfo(NULL, ..., AI_PASSIVE)`:
|
||||
|
||||
| Server build / Linux setting | Binding behavior |
|
||||
| --- | --- |
|
||||
| IPv6 enabled, `net.ipv6.bindv6only=0` | Requests `IPV6_V6ONLY=1` on the IPv6 socket, binds `[::]:PORT`, and separately binds `0.0.0.0:PORT`. It overrides the system's dual-stack default. |
|
||||
| IPv6 enabled, `net.ipv6.bindv6only=1` | The same explicit socket option and separate IPv4/IPv6 binds. |
|
||||
| `HAVE_IPV6` undefined | Skips every address family except `AF_INET`; only the IPv4 wildcard is attempted. |
|
||||
|
||||
`udp_server()` uses `wait_for_connection()`, which puts **every listener**
|
||||
from that list into `select()` and returns a readable socket for `recvfrom()`;
|
||||
it does not permanently choose one socket based on `getaddrinfo()` order.
|
||||
|
||||
The first two cases were traced with the actual GnuTLS 3.8.13 binary in
|
||||
separate Linux network namespaces: `setsockopt(IPV6_V6ONLY, [1])` and both
|
||||
UDP binds returned success under each setting. The no-IPv6 case was checked
|
||||
in source, not by building a second binary. The same bind implementation
|
||||
was checked directly on GitLab master.
|
||||
|
||||
Thus, successful normal startup for this invocation provides an explicit
|
||||
IPv4 socket; a lone IPv6 wildcard is not the expected success path.
|
||||
There is one portability caveat: upstream discards the return value of
|
||||
`setsockopt(IPV6_V6ONLY)`. On a platform where that call fails and the server
|
||||
ends up with only a dual-stack socket, this helper would time out despite
|
||||
IPv4 reachability. Such a platform needs additional handling before this
|
||||
patch can claim support. Blindly accepting every IPv6 wildcard would also
|
||||
accept IPv6-only sockets before the separate IPv4 bind finishes.
|
||||
|
||||
Source: [`src/serv.c`, `listen_socket()`](https://gitlab.com/gnutls/gnutls/-/blob/master/src/serv.c#L937),
|
||||
[`src/udp-serv.c`](https://gitlab.com/gnutls/gnutls/-/blob/master/src/udp-serv.c),
|
||||
and [`tests/serv-udp.sh`](https://gitlab.com/gnutls/gnutls/-/blob/master/tests/serv-udp.sh).
|
||||
|
||||
## Port matching and ownership limit
|
||||
|
||||
Only `-an` is passed to the socket-listing tool: BSD `netstat -u` selects
|
||||
Unix-domain sockets, whereas Linux `netstat -u` selects UDP. The parser
|
||||
handles the extra state column in `ss`, Linux colon-separated endpoints,
|
||||
and BSD dot-separated endpoints, including `*.PORT`. It matches the full
|
||||
local port and rejects TCP, IPv6 entries, peer ports, and longer numbers.
|
||||
|
||||
A live PID plus a bound port does **not** prove that PID owns the socket.
|
||||
Existing `GETPORT` selection checks for an unused port and uses a test
|
||||
port-lock directory; `launch_bare_server()` also calls
|
||||
`wait_for_free_port()` before starting the process. These are advisory:
|
||||
the launcher does not enforce the latter's result, and another process
|
||||
can bind between the check and launch. The patch does not close that race
|
||||
or add nonportable PID parsing. An unrelated process can satisfy the
|
||||
socket check; the real handshakes remain the functional check and may
|
||||
fail (or reach the wrong server). This is a startup-order fix, not a
|
||||
socket-ownership guarantee.
|
||||
|
||||
## Reproduction and focused checks
|
||||
|
||||
Apply the patch to an unpacked source tree, then run the companion checks
|
||||
with Python's standard library and a shell:
|
||||
|
||||
```sh
|
||||
patch --fuzz=0 -d /path/to/gnutls -p1 < udp-server-readiness.patch
|
||||
SHELL=/bin/sh python3 verify-readiness.py /path/to/gnutls/tests/scripts/common.sh -v
|
||||
```
|
||||
|
||||
Set `NETSTAT=/path/to/netstat` to exercise one outside `PATH`. The checks
|
||||
cover Linux/BSD output samples, false matches, immediate readiness,
|
||||
missing tools, process exit, timeout cleanup, and real IPv4 UDP sockets
|
||||
whose bind is delayed six seconds. The missing-tools fixture is skipped
|
||||
if an absolute fallback `ss` path cannot be hidden with `PATH`. Native
|
||||
BSD execution remains untested.
|
||||
|
||||
To reproduce with GnuTLS itself, run `tests/serv-udp.sh` with `SERV` pointing
|
||||
to a wrapper that sleeps six seconds, then `exec`s `gnutls-serv` with all
|
||||
arguments. Set `CLI` to the matching `gnutls-cli`, `srcdir` to the source
|
||||
`tests` directory, and `abs_top_builddir` to a writable build directory.
|
||||
With GnuTLS 3.8.13, the original helper failed the first handshake with
|
||||
`Connection refused`; the patched helper passed both with the same binaries.
|
||||
|
||||
## GnuTLS submission
|
||||
|
||||
Development and merge requests are on [GitLab](https://gitlab.com/gnutls/gnutls).
|
||||
[`CONTRIBUTING.md` on master](https://gitlab.com/gnutls/gnutls/-/blob/master/CONTRIBUTING.md)
|
||||
was read directly for this review. It requires the contributor's DCO
|
||||
`Signed-off-by`, successful and failure test coverage, consistent coding
|
||||
style, and adequate documentation; GitLab CI runs for merge requests.
|
||||
Its commenting guidance asks for comments explaining non-obvious behavior
|
||||
or protocol expectations. It does not prescribe an additional special
|
||||
test-suite comment. The patch now explains its IPv4 binding assumption
|
||||
next to the probe.
|
||||
|
||||
The submission will contain the shell patch, without the Python verifier
|
||||
or a new Python test dependency. The existing `serv-udp.sh` supplies the
|
||||
functional success check. Running it through the six-second startup
|
||||
wrapper supplies a reproducible regression case: it fails before the fix
|
||||
and passes after it. The local verifier was used to validate socket-output
|
||||
parsing and the helper's success, process-exit, skip-cleanup, and timeout
|
||||
branches. Those branch checks are local evidence, not new automated
|
||||
coverage in the upstream suite; the MR must state that distinction.
|
||||
|
||||
No dedicated unit-test harness for these shell helpers was found in the
|
||||
3.8.13 tests inspected. That does not establish that Python cannot be used
|
||||
upstream; keeping this submission dependency-free is a scope choice. Use
|
||||
the existing test and before/after reproduction as the submission's
|
||||
coverage argument, retaining the platform limitations above. Apply the
|
||||
patch in an upstream checkout and include those results with the
|
||||
contributor's own sign-off. No MR or sign-off has been created.
|
||||
|
||||
## Local NixOS integration and build results
|
||||
|
||||
`overlays/default.nix` imports the `overlays/patches` overlay, which loads
|
||||
`gnutls/default.nix` to apply the patch and keep `serv-udp.sh` enabled.
|
||||
The patch itself has no Nix dependencies and applies to 3.8.13 and GitLab
|
||||
master without fuzz.
|
||||
|
||||
The final patch was rebuilt with:
|
||||
|
||||
```sh
|
||||
nix build --no-link -L .#nixosConfigurations.jeeves.pkgs.gnutls
|
||||
```
|
||||
|
||||
That x86-64-v3 build passed: 927 tests, 796 passes, 131 existing skips,
|
||||
zero failures/errors, and `PASS: serv-udp.sh`. The patch bytes in the built
|
||||
derivation were compared with the repository artifact; both have SHA-256
|
||||
`59013d47fd446f2dd065012a2259ccc1898fedc8a053a630e13efa0076368760`.
|
||||
All seven local checks passed, including skip cleanup and exactly 90
|
||||
probes with 89 sleeps on timeout. The six-second before/after reproduction
|
||||
was also repeated successfully with the final helper.
|
||||
@@ -0,0 +1,6 @@
|
||||
{ gnutls }:
|
||||
gnutls.overrideAttrs (old: {
|
||||
# Keep the UDP handshake test enabled on loaded builders by waiting for
|
||||
# the server to bind its socket. Kept as a standalone patch for upstream.
|
||||
patches = (old.patches or [ ]) ++ [ ./udp-server-readiness.patch ];
|
||||
})
|
||||
@@ -0,0 +1,70 @@
|
||||
Subject: [PATCH] tests: wait for the UDP server socket before connecting
|
||||
|
||||
A fixed four-second sleep does not guarantee that gnutls-serv has bound
|
||||
its UDP socket on a busy builder. Poll the local IPv4 UDP endpoint using
|
||||
the existing ss/netstat discovery, with the same retry budget as the TCP
|
||||
helper. Fail early if the server exits, and retain the original handshake
|
||||
checks in serv-udp.sh.
|
||||
|
||||
Use flags common to ss and BSD/Linux netstat. Match the local endpoint
|
||||
and complete port number, excluding TCP, IPv6-only and peer endpoints.
|
||||
|
||||
--- a/tests/scripts/common.sh
|
||||
+++ b/tests/scripts/common.sh
|
||||
@@ -185,10 +185,55 @@
|
||||
fi
|
||||
}
|
||||
|
||||
+check_if_udp_port_bound() {
|
||||
+ local PORT=$1
|
||||
+ have_port_finder
|
||||
+ # Use only -an, which is shared by ss and BSD/Linux netstat. UDP has
|
||||
+ # no LISTEN state. Match the local IPv4 endpoint, not a peer port or
|
||||
+ # a longer port number. serv-udp.sh connects to 127.0.0.1;
|
||||
+ # listen_socket() in serv.c binds IPv4 separately and requests
|
||||
+ # IPV6_V6ONLY=1 for its IPv6 socket.
|
||||
+ $PFCMD -an | awk -v port="$PORT" '
|
||||
+ $1 == "udp" || $1 == "udp4" {
|
||||
+ # ss includes a state column; netstat does not.
|
||||
+ address = ($2 == "UNCONN" || $2 == "ESTAB") ? $5 : $4
|
||||
+ if (address ~ ("^[0-9.]+[.:]" port "$") ||
|
||||
+ address == "*." port)
|
||||
+ found = 1
|
||||
+ }
|
||||
+ END { exit !found }
|
||||
+ '
|
||||
+}
|
||||
+
|
||||
wait_udp_server() {
|
||||
local PID=$1
|
||||
+ local ret
|
||||
trap "test -n \"${PID}\" && kill ${PID};exit 1" 1 15 2
|
||||
- sleep 4
|
||||
+ local i=0
|
||||
+ # Use the same retry budget as wait_for_port(), but also stop if the
|
||||
+ # server exits before binding its socket.
|
||||
+ while test $i -lt 90; do
|
||||
+ if ! kill -0 "$PID" 2>/dev/null; then
|
||||
+ fail "" "UDP server $PID exited before binding port $PORT"
|
||||
+ fi
|
||||
+ # Contain have_port_finder's exit so a skip also stops the server.
|
||||
+ if (check_if_udp_port_bound "$PORT"); then
|
||||
+ return 0
|
||||
+ else
|
||||
+ ret=$?
|
||||
+ if test "$ret" = 77; then
|
||||
+ kill "$PID" 2>/dev/null || :
|
||||
+ wait "$PID" 2>/dev/null || :
|
||||
+ exit 77
|
||||
+ fi
|
||||
+ fi
|
||||
+ i=$((i + 1))
|
||||
+ if test $i -lt 90; then
|
||||
+ echo "try $i: waiting for UDP port $PORT"
|
||||
+ sleep 2
|
||||
+ fi
|
||||
+ done
|
||||
+ fail "$PID" "UDP server $PORT did not come up"
|
||||
}
|
||||
|
||||
create_testdir() {
|
||||
@@ -0,0 +1,180 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Exercise patched common.sh without building GnuTLS (Python standard library only).
|
||||
|
||||
Usage: python3 verify-readiness.py /path/to/patched/tests/scripts/common.sh
|
||||
Set SHELL to test another shell, and NETSTAT to test a netstat outside PATH.
|
||||
"""
|
||||
|
||||
# Use unittest so this upstream companion tool needs no pytest installation.
|
||||
# ruff: noqa: PT009
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
COMMON = str(Path(sys.argv.pop(1)).resolve())
|
||||
SHELL = os.environ.get("SHELL", "/bin/sh")
|
||||
|
||||
|
||||
class ReadinessTests(unittest.TestCase):
|
||||
"""Check endpoint parsing and the server startup lifecycle."""
|
||||
|
||||
def setUp(self) -> None:
|
||||
"""Create a socket-listing fixture for each check."""
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.tmp.cleanup)
|
||||
self.root = Path(self.tmp.name)
|
||||
self.fixture = self.root / "sockets"
|
||||
self.fixture.write_text("")
|
||||
self.finder = self.root / "port-finder"
|
||||
self.finder.write_text('#!/bin/sh\ncat "$SOCKET_FIXTURE"\n')
|
||||
self.finder.chmod(0o755)
|
||||
|
||||
def run_shell(self, body: str, **env: str) -> subprocess.CompletedProcess[str]:
|
||||
"""Source the actual helper and run a shell scenario."""
|
||||
return subprocess.run(
|
||||
[SHELL, "-c", '. "$COMMON"\n' + body],
|
||||
env={
|
||||
**os.environ,
|
||||
"COMMON": COMMON,
|
||||
"SOCKET_FIXTURE": str(self.fixture),
|
||||
"PFCMD": str(self.finder),
|
||||
"PORT": "12345",
|
||||
**env,
|
||||
},
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=20,
|
||||
check=False,
|
||||
)
|
||||
|
||||
def test_socket_formats_and_false_matches(self) -> None:
|
||||
"""Accept IPv4 UDP local endpoints and reject unrelated sockets."""
|
||||
cases = [
|
||||
("udp UNCONN 0 0 0.0.0.0:12345 0.0.0.0:*", True),
|
||||
("udp UNCONN 0 0 127.0.0.1:12345 0.0.0.0:*", True),
|
||||
("udp 0 0 0.0.0.0:12345 0.0.0.0:*", True),
|
||||
("udp4 0 0 *.12345 *.*", True),
|
||||
("udp 0 0 127.0.0.1.12345 *.*", True),
|
||||
("udp 0 0 *.12345 *.*", True),
|
||||
("udp UNCONN 0 0 0.0.0.0:123456 0.0.0.0:*", False),
|
||||
("udp 0 0 0.0.0.0:123456 0.0.0.0:*", False),
|
||||
("udp ESTAB 0 0 127.0.0.1:54321 127.0.0.1:12345", False),
|
||||
("udp 0 0 127.0.0.1:54321 127.0.0.1:12345", False),
|
||||
("tcp LISTEN 0 128 0.0.0.0:12345 0.0.0.0:*", False),
|
||||
("tcp 0 0 0.0.0.0:12345 0.0.0.0:* LISTEN", False),
|
||||
("udp UNCONN 0 0 [::]:12345 [::]:*", False),
|
||||
("udp UNCONN 0 0 *:12345 *:*", False),
|
||||
("udp6 0 0 :::12345 :::*", False),
|
||||
("udp6 0 0 *.12345 *.*", False),
|
||||
("", False),
|
||||
]
|
||||
for row, ready in cases:
|
||||
with self.subTest(row=row):
|
||||
self.fixture.write_text(row + "\n")
|
||||
result = self.run_shell('check_if_udp_port_bound "$PORT"')
|
||||
self.assertEqual(result.returncode, 0 if ready else 1, result.stderr)
|
||||
|
||||
def test_exited_server_fails_immediately(self) -> None:
|
||||
"""Fail without sleeping when the server has already exited."""
|
||||
result = self.run_shell(
|
||||
'true &\npid=$!\nwait "$pid"\nsleep() { echo "unexpected sleep" >&2; }\nwait_udp_server "$pid"'
|
||||
)
|
||||
self.assertEqual(result.returncode, 1)
|
||||
self.assertIn("exited before binding", result.stderr)
|
||||
self.assertNotIn("unexpected sleep", result.stderr)
|
||||
|
||||
def test_ready_socket_does_not_sleep(self) -> None:
|
||||
"""Check readiness before the first sleep."""
|
||||
self.fixture.write_text("udp UNCONN 0 0 0.0.0.0:12345 0.0.0.0:*\n")
|
||||
result = self.run_shell('sleep() { echo "unexpected sleep" >&2; }\nwait_udp_server "$$"')
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertNotIn("unexpected sleep", result.stderr)
|
||||
|
||||
def test_missing_port_finders_skip(self) -> None:
|
||||
"""Skip and stop the live server when no finder is available."""
|
||||
# have_port_finder also tries these paths independently of PATH.
|
||||
if any(os.access(f"{directory}/ss", os.X_OK) for directory in ("/sbin", "/usr/sbin", "/usr/local/sbin")):
|
||||
self.skipTest("an absolute ss path cannot be hidden by this PATH-only fixture")
|
||||
with subprocess.Popen(["sleep", "60"]) as server:
|
||||
try:
|
||||
result = self.run_shell(
|
||||
'unset PFCMD\nPATH=/nonexistent\nwait_udp_server "$SERVER_PID"',
|
||||
SERVER_PID=str(server.pid),
|
||||
)
|
||||
self.assertEqual(result.returncode, 77)
|
||||
self.assertIn("neither ss nor netstat found", result.stderr)
|
||||
server.wait(timeout=3)
|
||||
self.assertLess(server.returncode, 0)
|
||||
finally:
|
||||
if server.poll() is None:
|
||||
server.kill()
|
||||
|
||||
def test_timeout_is_bounded_and_cleans_up(self) -> None:
|
||||
"""Stop polling after the retry budget and terminate the server."""
|
||||
# Only accelerate the polling delay; keep a real live server process.
|
||||
self.finder.write_text('#!/bin/sh\necho probe >&2\ncat "$SOCKET_FIXTURE"\n')
|
||||
with subprocess.Popen(["sleep", "60"]) as server:
|
||||
try:
|
||||
result = self.run_shell(
|
||||
'sleep() { echo polling-sleep; }\nwait_udp_server "$SERVER_PID"',
|
||||
SERVER_PID=str(server.pid),
|
||||
)
|
||||
self.assertEqual(result.returncode, 1)
|
||||
self.assertIn("did not come up", result.stderr)
|
||||
self.assertEqual(result.stderr.count("probe\n"), 90)
|
||||
self.assertEqual(result.stdout.count("polling-sleep"), 89)
|
||||
server.wait(timeout=3)
|
||||
self.assertLess(server.returncode, 0)
|
||||
finally:
|
||||
if server.poll() is None:
|
||||
server.kill()
|
||||
|
||||
def test_server_exits_while_waiting(self) -> None:
|
||||
"""Detect a startup failure that happens after polling begins."""
|
||||
result = self.run_shell('sleep 1 &\npid=$!\nwait_udp_server "$pid"')
|
||||
self.assertEqual(result.returncode, 1)
|
||||
self.assertIn("exited before binding", result.stderr)
|
||||
self.assertIn("waiting for UDP port", result.stdout)
|
||||
|
||||
def test_real_socket_delayed_beyond_four_seconds(self) -> None:
|
||||
"""Wait for a real delayed bind with each installed port finder."""
|
||||
finders = [shutil.which("ss"), os.environ.get("NETSTAT") or shutil.which("netstat")]
|
||||
finders = [finder for finder in finders if finder]
|
||||
if not finders:
|
||||
self.skipTest("neither ss nor netstat available")
|
||||
for finder in finders:
|
||||
with self.subTest(finder=finder):
|
||||
with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as sock:
|
||||
sock.bind(("127.0.0.1", 0))
|
||||
port = sock.getsockname()[1]
|
||||
code = (
|
||||
"import socket,time,sys; time.sleep(6); "
|
||||
"s=socket.socket(socket.AF_INET,socket.SOCK_DGRAM); "
|
||||
"s.bind(('127.0.0.1',int(sys.argv[1]))); time.sleep(30)"
|
||||
)
|
||||
with subprocess.Popen([sys.executable, "-c", code, str(port)]) as server:
|
||||
try:
|
||||
started = time.monotonic()
|
||||
result = self.run_shell(
|
||||
'wait_udp_server "$SERVER_PID"',
|
||||
SERVER_PID=str(server.pid),
|
||||
PORT=str(port),
|
||||
PFCMD=finder,
|
||||
)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertGreaterEqual(time.monotonic() - started, 6)
|
||||
self.assertIsNone(server.poll())
|
||||
finally:
|
||||
server.terminate()
|
||||
server.wait(timeout=3)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,66 @@
|
||||
# Prometheus complete test parsing
|
||||
|
||||
CodeMirror gives editor-state creation a 20 ms synchronous parsing budget.
|
||||
The shared `createEditorState()` test helper can therefore return an
|
||||
incomplete syntax tree when the process is descheduled. The completion and
|
||||
vector-matching tests immediately inspect that tree.
|
||||
|
||||
## Scope and behavior
|
||||
|
||||
`complete-test-parsing.patch` changes only
|
||||
`module/codemirror-promql/src/test/utils-test.ts` inside `web/ui`. It completes
|
||||
the small test expression with `ensureSyntaxTree(..., Infinity)` and publishes
|
||||
the completed parse through an empty transaction so `syntaxTree(state)` sees
|
||||
it. Failure to obtain a tree raises an error.
|
||||
|
||||
The original assertions remain enabled, including `autocomplete topk params 2`
|
||||
and `foo * on(test,blub) bar`. The unlimited budget applies to the test helper;
|
||||
production editor parsing budgets are unchanged.
|
||||
|
||||
## Reproduction and focused checks
|
||||
|
||||
Use a disposable Prometheus 3.14.0 checkout. The patch root is `web/ui`, matching
|
||||
the Nix assets derivation. From this directory:
|
||||
|
||||
```sh
|
||||
patch --fuzz=0 -d /path/to/prometheus/web/ui -p1 < complete-test-parsing.patch
|
||||
cd /path/to/prometheus/web/ui
|
||||
pnpm install --frozen-lockfile
|
||||
pnpm --filter @prometheus-io/lezer-promql build
|
||||
pnpm --filter @prometheus-io/codemirror-promql test
|
||||
```
|
||||
|
||||
To force the scheduling condition, temporarily append this clock to
|
||||
`module/codemirror-promql/setupJest.cjs` in the disposable checkout:
|
||||
|
||||
```js
|
||||
let parseClock = 0;
|
||||
Date.now = () => (parseClock += 25);
|
||||
```
|
||||
|
||||
Each clock read crosses the editor's initial parsing budget. Against the
|
||||
original helper, the hybrid and vector suites have 186 failures, including
|
||||
both locally excluded cases. With the patch, all 386 CodeMirror tests pass
|
||||
under that same clock. Remove the injected clock before normal builds.
|
||||
|
||||
## Upstream status
|
||||
|
||||
This is a standalone test-helper patch for Prometheus 3.14.0. No upstream
|
||||
submission was made during this work. Recheck the helper when updating
|
||||
Prometheus or CodeMirror, including how an ensured parse becomes visible
|
||||
through the editor state.
|
||||
|
||||
## Local NixOS integration and build results
|
||||
|
||||
[`../default.nix`](../default.nix) loads `default.nix`, which patches the
|
||||
separate assets derivation. It updates both `passthru.assets` and the main
|
||||
Prometheus build's reference to those assets. From the repository root:
|
||||
|
||||
```sh
|
||||
nix build --no-link -L .#nixosConfigurations.jeeves.pkgs.prometheus.assets
|
||||
```
|
||||
|
||||
The full x86-64-v3 assets build passed with the normal clock, including the
|
||||
CodeMirror and UI suites. Host-flake evaluation confirmed that the main
|
||||
Prometheus derivation refers to these patched assets. The Go server package
|
||||
was not rebuilt for this test-helper change.
|
||||
@@ -0,0 +1,36 @@
|
||||
Subject: [PATCH] tests: finish parsing before inspecting editor state
|
||||
|
||||
EditorState creation has a 20 ms parsing budget. A descheduled test can
|
||||
therefore observe an incomplete tree. Finish these small test documents
|
||||
without an interactive deadline and publish the result with a transaction.
|
||||
Keep the original completion and vector-matching assertions enabled.
|
||||
|
||||
--- a/module/codemirror-promql/src/test/utils-test.ts
|
||||
+++ b/module/codemirror-promql/src/test/utils-test.ts
|
||||
@@ -13,7 +13,7 @@
|
||||
|
||||
import { parser } from '@prometheus-io/lezer-promql';
|
||||
import { EditorState } from '@codemirror/state';
|
||||
-import { LRLanguage } from '@codemirror/language';
|
||||
+import { ensureSyntaxTree, LRLanguage } from '@codemirror/language';
|
||||
import nock from 'nock';
|
||||
import path from 'path';
|
||||
import { fileURLToPath } from 'url';
|
||||
@@ -23,10 +23,16 @@
|
||||
const __dirname = path.dirname(fileURLToPath(import.meta.url));
|
||||
|
||||
export function createEditorState(expr: string): EditorState {
|
||||
- return EditorState.create({
|
||||
+ const state = EditorState.create({
|
||||
doc: expr,
|
||||
extensions: lightPromQLSyntax,
|
||||
});
|
||||
+ // These tests need a complete tree, independent of the editor's time budget.
|
||||
+ if (!ensureSyntaxTree(state, state.doc.length, Infinity)) {
|
||||
+ throw new Error('Unable to parse the test expression');
|
||||
+ }
|
||||
+ // Publish the completed parse so syntaxTree(state) sees it too.
|
||||
+ return state.update({}).state;
|
||||
}
|
||||
|
||||
export function mockPrometheusServer(): void {
|
||||
@@ -0,0 +1,17 @@
|
||||
{ prometheus }:
|
||||
prometheus.overrideAttrs (
|
||||
old:
|
||||
let
|
||||
assets = old.passthru.assets.overrideAttrs (assetsOld: {
|
||||
patches = (assetsOld.patches or [ ]) ++ [ ./complete-test-parsing.patch ];
|
||||
});
|
||||
in
|
||||
{
|
||||
postPatch = builtins.replaceStrings [ "${old.passthru.assets}" ] [ "${assets}" ] (
|
||||
builtins.unsafeDiscardStringContext old.postPatch
|
||||
);
|
||||
passthru = old.passthru // {
|
||||
inherit assets;
|
||||
};
|
||||
}
|
||||
)
|
||||
@@ -0,0 +1,79 @@
|
||||
# pytest-xdist test fixes
|
||||
|
||||
With two workers and a restart limit of three, the fourth worker crash
|
||||
requests shutdown while another test can still be running. That test may
|
||||
also crash. The original queued-work test requires exactly four failures,
|
||||
even though five failures can occur without exceeding the replacement limit.
|
||||
|
||||
## Scope and behavior
|
||||
|
||||
`concurrent-worker-crashes.patch` changes the assertions in
|
||||
`TestNodeFailure.test_max_worker_restart_tests_queued` in
|
||||
`testing/acceptance_test.py`. It requires exactly three replacements, four or
|
||||
five failed tests, the failed-tests exit status, the limit message, and no
|
||||
internal error. It retains the two-worker workload and ten queued tests.
|
||||
|
||||
`worker-startup-timeout.patch` changes the remote-test helper's event timeout
|
||||
from 10 to 60 seconds so loaded builders have time to start workers. The
|
||||
helper returns immediately when an event arrives and still has a bounded wait.
|
||||
|
||||
The existing nixpkgs pytest-9 compatibility patches remain in place.
|
||||
Production scheduling and worker-restart behavior are unchanged.
|
||||
|
||||
## Reproduction and focused checks
|
||||
|
||||
Use a disposable pytest-xdist 3.8.0 checkout with its test dependencies and
|
||||
the nixpkgs pytest-9 compatibility patches where required. From this directory:
|
||||
|
||||
```sh
|
||||
patch --fuzz=0 -d /path/to/pytest-xdist -p1 < concurrent-worker-crashes.patch
|
||||
patch --fuzz=0 -d /path/to/pytest-xdist -p1 < worker-startup-timeout.patch
|
||||
cd /path/to/pytest-xdist
|
||||
python -m pytest testing/acceptance_test.py \
|
||||
-k test_max_worker_restart_tests_queued -q
|
||||
python -m pytest testing/test_remote.py -q
|
||||
```
|
||||
|
||||
Twenty unmodified runs passed during the review. To force the failing
|
||||
schedule, modify the generated crashing test in a disposable checkout to
|
||||
accept `worker_id`: make `gw3` wait for a marker created by `gw4`, and make
|
||||
`gw4` pause 0.1 seconds after creating the marker. Then both have in-flight
|
||||
tests when shutdown starts. Bound the marker wait so a reproduction failure
|
||||
cannot hang the suite. The original assertion fails on five reported
|
||||
failures; the patched test passes.
|
||||
|
||||
## Worker startup and outer concurrency
|
||||
|
||||
[`default.nix`](default.nix) runs the outer suite with one worker to limit
|
||||
nested process pools. This is a Nix test-runner setting; the source timeout
|
||||
change lives in [`worker-startup-timeout.patch`](worker-startup-timeout.patch).
|
||||
|
||||
A separate reproduction inserts an 11-second `pytest_sessionstart` delay
|
||||
into the child created by `test_basic_collect_and_runtests` in
|
||||
`testing/test_remote.py`. The original 10-second channel wait fails; the
|
||||
60-second wait passes. This bounds waits for test worker events, including
|
||||
startup, rather than changing a product deadline.
|
||||
|
||||
## Upstream status
|
||||
|
||||
These are standalone test patches for pytest-xdist 3.8.0. No upstream submission
|
||||
was made during this work. Recheck the allowed in-flight failures, replacement
|
||||
count, and remote-test wait when updating the scheduler or worker behavior.
|
||||
|
||||
## Local NixOS integration and build results
|
||||
|
||||
[`../default.nix`](../default.nix) loads `default.nix` through
|
||||
`pythonPackagesExtensions`. From the repository root:
|
||||
|
||||
```sh
|
||||
nix build --no-link -L .#nixosConfigurations.jeeves.pkgs.python314Packages.pytest-xdist
|
||||
```
|
||||
|
||||
After consolidating the settings in this directory, the full x86-64-v3 package
|
||||
build passed 185 tests, with 6 existing skips and 10 expected failures. Nix
|
||||
evaluation confirmed the same outer-worker limit and preserved existing
|
||||
patches, with the timeout now applied as a source patch.
|
||||
|
||||
The earlier forced concurrent-crash and delayed-startup reproductions passed
|
||||
after their fixes; the focused crash test also passed after formatting its
|
||||
assertion.
|
||||
@@ -0,0 +1,29 @@
|
||||
Subject: [PATCH] tests: count replacements when checking the worker restart limit
|
||||
|
||||
With two workers, another in-flight test may crash after the fourth
|
||||
crash requests shutdown. Either four or five failed tests is valid.
|
||||
Require exactly three replacements and the failed-tests exit status,
|
||||
while preserving the queued-work and no-internal-error assertions.
|
||||
|
||||
--- a/testing/acceptance_test.py
|
||||
+++ b/testing/acceptance_test.py
|
||||
@@ -1011,9 +1011,18 @@
|
||||
"worker*crashed while running*",
|
||||
"worker*crashed while running*",
|
||||
"* xdist: maximum crashed workers reached: 3 *",
|
||||
- "* 4 failed in *",
|
||||
]
|
||||
)
|
||||
+ # A second in-flight test may crash after shutdown is requested.
|
||||
+ # The restart limit constrains replacements, not concurrent failures.
|
||||
+ replacements = sum(
|
||||
+ line.startswith("replacing crashed worker ") for line in res.stdout.lines
|
||||
+ )
|
||||
+ assert replacements == 3
|
||||
+ failed = res.parseoutcomes()["failed"]
|
||||
+ assert failed in (4, 5)
|
||||
+ res.assert_outcomes(failed=failed)
|
||||
+ assert res.ret == pytest.ExitCode.TESTS_FAILED
|
||||
assert "INTERNALERROR" not in res.stdout.str()
|
||||
|
||||
def test_max_worker_restart_die(self, pytester: pytest.Pytester) -> None:
|
||||
@@ -0,0 +1,12 @@
|
||||
{ pytest-xdist }:
|
||||
pytest-xdist.overridePythonAttrs (old: {
|
||||
patches = (old.patches or [ ]) ++ [
|
||||
./concurrent-worker-crashes.patch
|
||||
./worker-startup-timeout.patch
|
||||
];
|
||||
|
||||
# The suite exercises its own worker pools. Limit the outer suite to one worker.
|
||||
preCheck = builtins.replaceStrings [ "--numprocesses=$NIX_BUILD_CORES" ] [ "--numprocesses=1" ] (
|
||||
old.preCheck or ""
|
||||
);
|
||||
})
|
||||
@@ -0,0 +1,19 @@
|
||||
Subject: [PATCH] tests: allow more time for remote worker events
|
||||
|
||||
Worker startup can exceed ten seconds on heavily loaded builders. Allow
|
||||
the remote-test helper to wait up to sixty seconds for worker events.
|
||||
The wait still returns as soon as an event arrives and remains bounded.
|
||||
Production worker timeouts and test assertions are unchanged.
|
||||
|
||||
--- a/testing/test_remote.py
|
||||
+++ b/testing/test_remote.py
|
||||
@@ -17,7 +17,8 @@
|
||||
from xdist.workermanage import WorkerController
|
||||
|
||||
|
||||
-WAIT_TIMEOUT = 10.0
|
||||
+# Allow worker events extra time on heavily loaded builders.
|
||||
+WAIT_TIMEOUT = 60.0
|
||||
|
||||
|
||||
def check_marshallable(d: object) -> None:
|
||||
@@ -0,0 +1,70 @@
|
||||
# SciPy STFT test tolerances
|
||||
|
||||
The x86-64-v3 build can produce small floating-point residuals in inverse-STFT
|
||||
comparisons and scaling round trips. The original bounds reject these results,
|
||||
including residuals around `4e-17` where a round trip expects zero for a signal
|
||||
with amplitude 2.
|
||||
|
||||
## Scope and behavior
|
||||
|
||||
`stft-test-tolerances.patch` changes only the signal tests:
|
||||
|
||||
- The inverse-STFT comparison in `_scipy_spectral_test_shim.py` uses
|
||||
`max(1e-7, 2 * np.finfo(x.dtype).eps)` as its relative tolerance. Float64
|
||||
keeps the original bound, and the existing i686 override remains.
|
||||
- Three scaling round trips in `test_spectral.py` gain an absolute tolerance
|
||||
of one epsilon for the input dtype, allowing small residuals near zero.
|
||||
|
||||
The tests remain enabled, and the production STFT implementation is unchanged.
|
||||
|
||||
## Reproduction and focused checks
|
||||
|
||||
From this directory, apply the patch to a disposable SciPy 1.18.0 checkout:
|
||||
|
||||
```sh
|
||||
patch --fuzz=0 -d /path/to/scipy -p1 < stft-test-tolerances.patch
|
||||
```
|
||||
|
||||
Build and install that tree with SciPy's test dependencies. From outside the
|
||||
source directory, run the installed tests:
|
||||
|
||||
```sh
|
||||
python -m pytest --pyargs scipy.signal.tests.test_spectral \
|
||||
-k 'roundtrip_float32 or roundtrip_scaling' -q
|
||||
```
|
||||
|
||||
Use the same compiler flags and numerical libraries for before/after runs.
|
||||
The earlier reproduction called `TestSTFT.test_roundtrip_float32` and
|
||||
`TestSTFT.test_roundtrip_scaling` against the x86-64-v3 libraries, then loaded
|
||||
patched copies of the test modules. Both failed with the original bounds
|
||||
and passed with the adjusted bounds.
|
||||
|
||||
## Upstream status
|
||||
|
||||
[SciPy issue #25488](https://github.com/scipy/scipy/issues/25488) records
|
||||
related test failures with architecture-specific compiler flags. It is
|
||||
context for the local tolerance repair; this exact patch has not been
|
||||
submitted upstream during this work.
|
||||
|
||||
## Local NixOS integration and build results
|
||||
|
||||
[`../default.nix`](../default.nix) loads [`default.nix`](default.nix) through
|
||||
`pythonPackagesExtensions`, preserving the package's existing patches.
|
||||
The override also covers SciPy used to test other Python dependencies,
|
||||
including pgvector in portal's shared Python environment.
|
||||
|
||||
From the repository root:
|
||||
|
||||
```sh
|
||||
nix build --no-link -L .#nixosConfigurations.portal-1.pkgs.python314Packages.scipy
|
||||
```
|
||||
|
||||
The original remote build of patched SciPy 1.18.0 passed 87,723 tests, with
|
||||
8,342 skips, 300 expected failures, and 22 unexpected passes. The patch and
|
||||
override have been restored byte-for-byte from commit `24cbf74f`; those counts
|
||||
describe the earlier full build.
|
||||
|
||||
Restoration checks confirmed that the patch applies to the pinned source
|
||||
without fuzz, portal's evaluated SciPy retains its existing patch and install
|
||||
checks, and pgvector uses the patched SciPy. A full package or system rebuild
|
||||
was not repeated for this restoration.
|
||||
@@ -0,0 +1,5 @@
|
||||
{ scipy }:
|
||||
scipy.overridePythonAttrs (old: {
|
||||
# Keep the STFT tests enabled with tolerances for x86-64-v3 rounding.
|
||||
patches = (old.patches or [ ]) ++ [ ./stft-test-tolerances.patch ];
|
||||
})
|
||||
@@ -0,0 +1,51 @@
|
||||
Subject: [PATCH] signal: allow floating-point rounding in STFT tests
|
||||
|
||||
Keep the STFT tests enabled for x86-64-v3 builds. Allow two float32
|
||||
epsilons of relative error when comparing inverse-STFT implementations;
|
||||
float64 and the existing i686 override remain unchanged. Allow one
|
||||
float64 epsilon of absolute error in all three scaling round trips,
|
||||
which otherwise require exact zeros (observed residual: 4e-17 for a
|
||||
signal with amplitude 2).
|
||||
|
||||
Upstream issue: https://github.com/scipy/scipy/issues/25488
|
||||
|
||||
--- a/scipy/signal/tests/_scipy_spectral_test_shim.py
|
||||
+++ b/scipy/signal/tests/_scipy_spectral_test_shim.py
|
||||
@@ -294,7 +294,7 @@
|
||||
|
||||
# Adapted tolerances to account for resolution loss:
|
||||
atol = np.finfo(x.dtype).resolution*2 # instead of default atol = 0
|
||||
- rtol = 1e-7 # default for np.allclose()
|
||||
+ rtol = max(1e-7, 2 * np.finfo(x.dtype).eps)
|
||||
|
||||
# Relax atol on 32-Bit platforms a bit to pass CI tests.
|
||||
# - Not clear why there are discrepancies (in the FFT maybe?)
|
||||
--- a/scipy/signal/tests/test_spectral.py
|
||||
+++ b/scipy/signal/tests/test_spectral.py
|
||||
@@ -2044,7 +2044,7 @@
|
||||
|
||||
# Test round trip:
|
||||
x1 = istft(Zs, boundary=True, scaling='spectrum')[1]
|
||||
- assert_allclose(x1, x)
|
||||
+ assert_allclose(x1, x, atol=np.finfo(x.dtype).eps)
|
||||
|
||||
# For a Hann-windowed 256 sample length FFT, we expect a peak at
|
||||
# frequency 64 (since it is 1/4 the length of X) with a height of 1
|
||||
@@ -2074,7 +2074,7 @@
|
||||
|
||||
# Test round trip:
|
||||
x1 = istft(Zp, input_onesided=False, boundary=True, scaling='psd')[1]
|
||||
- assert_allclose(x1, x)
|
||||
+ assert_allclose(x1, x, atol=np.finfo(x.dtype).eps)
|
||||
|
||||
# The power of the one-sided psd-scaled STFT can be determined
|
||||
# analogously (note that the two sides are not of equal shape):
|
||||
@@ -2094,7 +2094,7 @@
|
||||
|
||||
# Test round trip:
|
||||
x1 = istft(Zp0, input_onesided=True, boundary=True, scaling='psd')[1]
|
||||
- assert_allclose(x1, x)
|
||||
+ assert_allclose(x1, x, atol=np.finfo(x.dtype).eps)
|
||||
|
||||
|
||||
class TestSampledSpectralRepresentations:
|
||||
@@ -0,0 +1,58 @@
|
||||
# Sentry SDK thread-metadata test isolation
|
||||
|
||||
The fallback tests globally mock `threading.current_thread` while a worker
|
||||
is running. Python 3.14's `Thread.join()` also calls that function. A one-use
|
||||
mock can therefore be consumed by the wrong caller or raise `StopIteration`
|
||||
when the main thread joins the worker.
|
||||
|
||||
## Scope and behavior
|
||||
|
||||
`isolate-threading-mocks.patch` changes three neighboring thread-metadata
|
||||
tests in `tests/test_utils.py`, including the formerly excluded
|
||||
`test_get_current_thread_meta_main_thread`.
|
||||
|
||||
Each test replaces only `sentry_sdk.utils.threading`, wraps the real module
|
||||
for unmocked operations, and sets the SDK lookup's return value. The real
|
||||
`Thread.join()` continues using Python's unmodified `threading` module.
|
||||
The fallback-result assertions remain; SDK production code is unchanged.
|
||||
|
||||
## Reproduction and focused checks
|
||||
|
||||
Use a disposable Sentry SDK 2.66.0 checkout and its Python test dependencies.
|
||||
From this directory:
|
||||
|
||||
```sh
|
||||
patch --fuzz=0 -d /path/to/sentry-python -p1 < isolate-threading-mocks.patch
|
||||
cd /path/to/sentry-python
|
||||
python -m pytest tests/test_utils.py -k get_current_thread_meta -q
|
||||
```
|
||||
|
||||
To reproduce the race, hold the worker inside its mock just after
|
||||
`get_current_thread_meta()` returns, signal that point to the main thread,
|
||||
and call `Thread.join()` before releasing the worker. Use an independent
|
||||
bounded release so the patched join can finish. The original test raises
|
||||
`StopIteration` in `join`; the patched test passes under the same schedule.
|
||||
Perform this scheduling instrumentation only in a disposable checkout.
|
||||
|
||||
## Upstream status
|
||||
|
||||
This is a standalone test patch for Sentry SDK 2.66.0. No upstream submission
|
||||
was made during this work. Recheck mock isolation and Python threading
|
||||
behavior when upgrading the SDK or interpreter.
|
||||
|
||||
## Local NixOS integration and build results
|
||||
|
||||
[`../default.nix`](../default.nix) loads `default.nix` through
|
||||
`pythonPackagesExtensions`. From the repository root:
|
||||
|
||||
```sh
|
||||
nix build --no-link -L .#nixosConfigurations.jeeves.pkgs.python314Packages.sentry-sdk
|
||||
```
|
||||
|
||||
The patched package passed 2,356 tests with 116 existing skips on Python
|
||||
3.14.7. The controlled join reproduction failed before the fix and passed
|
||||
after it.
|
||||
|
||||
That package build used the preceding dependency set with this patch to avoid
|
||||
unrelated rebuilds after pytest-xdist changed. The integrated host derivation
|
||||
was evaluated; a complete NixOS rebuild was not performed.
|
||||
@@ -0,0 +1,4 @@
|
||||
{ sentry-sdk }:
|
||||
sentry-sdk.overridePythonAttrs (old: {
|
||||
patches = (old.patches or [ ]) ++ [ ./isolate-threading-mocks.patch ];
|
||||
})
|
||||
@@ -0,0 +1,41 @@
|
||||
Subject: [PATCH] tests: isolate SDK thread lookup mocks from Python threading
|
||||
|
||||
Thread.join also calls threading.current_thread on Python 3.14. A global
|
||||
single-use side effect can be consumed by join instead of the SDK, or
|
||||
raise StopIteration in join after the SDK consumes it. Patch the SDK's
|
||||
module binding and delegate unmocked operations to the real module.
|
||||
Apply the same isolation to the adjacent invalid-thread fallback tests.
|
||||
|
||||
--- a/tests/test_utils.py
|
||||
+++ b/tests/test_utils.py
|
||||
@@ -914,7 +914,8 @@
|
||||
results = Queue(maxsize=1)
|
||||
|
||||
def target():
|
||||
- with mock.patch("threading.current_thread", side_effect=["fake thread"]):
|
||||
+ with mock.patch("sentry_sdk.utils.threading", wraps=threading) as sdk_threading:
|
||||
+ sdk_threading.current_thread.return_value = "fake thread"
|
||||
results.put(get_current_thread_meta())
|
||||
|
||||
thread = threading.Thread(target=target)
|
||||
@@ -930,7 +931,9 @@
|
||||
|
||||
def target():
|
||||
# mock that somehow the current thread doesn't exist
|
||||
- with mock.patch("threading.current_thread", side_effect=[None]):
|
||||
+ # Keep the real threading module intact for concurrent Thread.join calls.
|
||||
+ with mock.patch("sentry_sdk.utils.threading", wraps=threading) as sdk_threading:
|
||||
+ sdk_threading.current_thread.return_value = None
|
||||
results.put(get_current_thread_meta())
|
||||
|
||||
main_thread = threading.main_thread()
|
||||
@@ -945,7 +948,8 @@
|
||||
results = Queue(maxsize=1)
|
||||
|
||||
def target():
|
||||
- with mock.patch("threading.current_thread", return_value="fake thread"):
|
||||
+ with mock.patch("sentry_sdk.utils.threading", wraps=threading) as sdk_threading:
|
||||
+ sdk_threading.current_thread.return_value = "fake thread"
|
||||
results.put(get_current_thread_meta())
|
||||
|
||||
main_thread = threading.main_thread()
|
||||
Reference in New Issue
Block a user