Compare commits
11
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
24af88be1d | ||
|
|
f7e26d9d07 | ||
|
|
b7dfe1d95b | ||
|
|
e038a44cb6 | ||
|
|
4ecc1fd752 | ||
|
|
fee4d31971 | ||
|
|
5b4bc4b72f | ||
|
|
a37b20979d | ||
|
|
af4a917d92 | ||
|
|
1cac244259 | ||
|
|
03d560eb10 |
@@ -8,8 +8,23 @@ on:
|
||||
- cron: "0 22 * * *"
|
||||
|
||||
jobs:
|
||||
prebuild-common:
|
||||
name: prebuild-common-x86-64-v3
|
||||
runs-on: nix-cache-builder
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
# portal-1 is the smallest system closure: 95% of its derivations are
|
||||
# shared by all five systems, so it is a maintainable common cache seed.
|
||||
# Keep going so one failing package does not stop unrelated cache entries
|
||||
# from being built.
|
||||
- name: Build common packages
|
||||
run: nixos-rebuild build --keep-going --accept-flake-config --flake ./#portal-1
|
||||
- name: Copy common packages to nix-cache
|
||||
run: nix copy --accept-flake-config --to unix:///host-nix/var/nix/daemon-socket/socket .#nixosConfigurations.portal-1.config.system.build.toplevel
|
||||
|
||||
build:
|
||||
name: build-${{ matrix.system }}
|
||||
needs: prebuild-common
|
||||
runs-on: self-hosted
|
||||
strategy:
|
||||
matrix:
|
||||
|
||||
+12
-4
@@ -1,13 +1,13 @@
|
||||
{ inputs, ... }:
|
||||
{
|
||||
# When applied, the stable nixpkgs set (declared in the flake inputs) will be accessible through 'pkgs.stable'
|
||||
# Additional package sets are accessible through `pkgs.<name>`.
|
||||
stable = final: _prev: {
|
||||
stable = import inputs.nixpkgs-stable {
|
||||
system = final.stdenv.hostPlatform.system;
|
||||
config.allowUnfree = true;
|
||||
};
|
||||
};
|
||||
# When applied, the master nixpkgs set (declared in the flake inputs) will be accessible through 'pkgs.master'
|
||||
|
||||
master = final: _prev: {
|
||||
master = import inputs.nixpkgs-master {
|
||||
system = final.stdenv.hostPlatform.system;
|
||||
@@ -15,6 +15,14 @@
|
||||
};
|
||||
};
|
||||
|
||||
test-exclusions = import ./test-exclusions.nix;
|
||||
x86-64-v3-workarounds = import ./x86-64-v3-workarounds.nix;
|
||||
# Baseline x86-64 (v1) packages for prebuilt applications that should not
|
||||
# inherit an x86-64-v3 host platform.
|
||||
x86-v1 = final: _prev: {
|
||||
x86-v1 = import inputs.nixpkgs {
|
||||
system = final.stdenv.hostPlatform.system;
|
||||
config.allowUnfree = true;
|
||||
};
|
||||
};
|
||||
|
||||
patches = import ./patches;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
# Package patches
|
||||
|
||||
Each package follows the [GnuTLS layout](gnutls/README.md):
|
||||
|
||||
- `default.nix` applies the patch through the package overlay.
|
||||
- A descriptive `.patch` file contains the standalone upstream source change.
|
||||
- `README.md` explains the problem, scope, reproduction, upstream status,
|
||||
Nix integration, and recorded validation limits.
|
||||
- Companion `verify-*` tools live beside the patch when needed; otherwise
|
||||
the README gives commands for the package's existing tests.
|
||||
|
||||
Keep package-specific evidence in its directory. Patch headers explain the
|
||||
change independently of Nix, and `default.nix` preserves existing patches.
|
||||
|
||||
| Package | Repair |
|
||||
| --- | --- |
|
||||
| [Abseil](abseil/README.md) | Public BMI2 header in Electron, Deno, and Signal's vendored copies |
|
||||
| [Backrefs](backrefs/README.md) | Match the regex timeout's CPU clock |
|
||||
| [GnuTLS](gnutls/README.md) | Wait for the UDP server socket before connecting |
|
||||
| [Jupyter Server](jupyter-server/README.md) | Exercise the correct shared future during reconnect |
|
||||
| [Prometheus](prometheus/README.md) | Complete parsing before inspecting the test editor state |
|
||||
| [pytest-xdist](pytest-xdist/README.md) | Check worker replacements and allow startup on loaded builders |
|
||||
| [SciPy](scipy/README.md) | Account for floating-point rounding in STFT tests |
|
||||
| [Sentry SDK](sentry-sdk/README.md) | Isolate SDK thread mocks from Python's threading module |
|
||||
| [Torchaudio](torchaudio/README.md) | Compare pitch-shift batches at appropriate precision |
|
||||
| [TorchCodec](torchcodec/README.md) | Match the reference MP3 encoder's sample format |
|
||||
|
||||
## Local NixOS integration
|
||||
|
||||
[`../default.nix`](../default.nix) imports this directory's
|
||||
[`default.nix`](default.nix), which wires each package's override into the
|
||||
package set. Abseil repairs several vendored copies and is gated on
|
||||
`x86-64-v3`; Prometheus patches its separate assets derivation; Python
|
||||
packages use `pythonPackagesExtensions`.
|
||||
|
||||
The [pytest-xdist directory](pytest-xdist/README.md) also owns its outer-worker
|
||||
limit and remote-worker event timeout. These package overrides add no skipped
|
||||
tests. Existing nixpkgs exclusions remain separate from these repairs.
|
||||
|
||||
The test-exclusion review used Python 3.14.7 and the pinned x86-64-v3 package
|
||||
set. Host-flake evaluation verified patch wiring, Python install checks,
|
||||
removal of the local skips, and Prometheus's reference to the patched assets.
|
||||
Jupyter and Sentry package tests used the preceding dependency set with the
|
||||
new package patch to avoid unrelated rebuilds after pytest-xdist changed.
|
||||
No complete NixOS rebuild was performed. Individual READMEs distinguish
|
||||
package builds, focused tests, and checks that have not been run.
|
||||
@@ -0,0 +1,59 @@
|
||||
# Abseil BMI2 public header
|
||||
|
||||
Vendored Abseil includes `bmi2intrin.h` directly when `__BMI2__` is enabled.
|
||||
Compilers reject that internal header without the umbrella-header setup.
|
||||
`bmi2-public-header.patch` includes `immintrin.h` instead, allowing builds
|
||||
that enable BMI2 through `-march=x86-64-v3`.
|
||||
|
||||
## Scope and behavior
|
||||
|
||||
The patch changes one include in
|
||||
`third_party/abseil-cpp/absl/container/internal/raw_hash_set.h`.
|
||||
`default.nix` applies it to Electron 43's unwrapped package, Deno's
|
||||
`librusty_v8`, and Signal's WebRTC dependency. It also supplies the patched
|
||||
Electron package to Signal. These overrides apply only to `x86-64-v3`.
|
||||
|
||||
The shared file path is relative to each vendoring project's source root,
|
||||
not the root of a standalone Abseil checkout. No hash-table algorithm or
|
||||
test exclusion changes.
|
||||
|
||||
## Reproduction and focused checks
|
||||
|
||||
From this directory, check and apply the patch to each vendored source tree:
|
||||
|
||||
```sh
|
||||
patch --dry-run --fuzz=0 -d /path/to/vendor-source -p1 < bmi2-public-header.patch
|
||||
patch --fuzz=0 -d /path/to/vendor-source -p1 < bmi2-public-header.patch
|
||||
```
|
||||
|
||||
A small compiler check isolates the header requirement. With GCC or Clang
|
||||
on x86-64, compile `#include <bmi2intrin.h>` using `-march=x86-64-v3`; the
|
||||
compiler rejects the direct include. Changing it to `#include <immintrin.h>`
|
||||
should compile. The full consumer builds below check integration with their
|
||||
actual toolchains.
|
||||
|
||||
## Upstream status
|
||||
|
||||
Abseil addressed this issue through
|
||||
[PR #2071](https://github.com/abseil/abseil-cpp/pull/2071), imported by its
|
||||
upstream workflow. That change uses `x86gprintrin.h`; this local variant uses
|
||||
the public `immintrin.h` umbrella header for the vendored toolchains.
|
||||
Keep the workaround until all three bundled copies include a compatible fix.
|
||||
This file is a local adaptation, not a verbatim copy of the upstream diff.
|
||||
|
||||
## Local NixOS integration and build results
|
||||
|
||||
[`../default.nix`](../default.nix) merges this directory's overlay fragment
|
||||
because it repairs multiple packages. From the repository root, the consumer
|
||||
build commands are:
|
||||
|
||||
```sh
|
||||
nix build --no-link -L .#nixosConfigurations.jeeves.pkgs.deno
|
||||
nix build --no-link -L .#nixosConfigurations.jeeves.pkgs.electron_43
|
||||
nix build --no-link -L .#nixosConfigurations.jeeves.pkgs.signal-desktop
|
||||
```
|
||||
|
||||
The earlier extraction checked the vendored header snapshots and evaluated
|
||||
all three patch attachments. Those records do not establish successful full
|
||||
consumer rebuilds. No new compiler or consumer build was run for the layout
|
||||
change; the patch and override are unchanged.
|
||||
@@ -0,0 +1,20 @@
|
||||
Subject: [PATCH] abseil: include BMI2 intrinsics through the public header
|
||||
|
||||
GCC and Clang reject direct inclusion of bmi2intrin.h. Include immintrin.h
|
||||
instead so that the compiler supplies the required intrinsic setup when
|
||||
BMI2 is enabled, including builds targeting x86-64-v3.
|
||||
|
||||
This patch is shared by the vendored Abseil copies in Electron, rusty_v8
|
||||
(Deno), and Signal's WebRTC build.
|
||||
|
||||
--- a/third_party/abseil-cpp/absl/container/internal/raw_hash_set.h
|
||||
+++ b/third_party/abseil-cpp/absl/container/internal/raw_hash_set.h
|
||||
@@ -226,7 +226,7 @@
|
||||
#endif
|
||||
|
||||
#ifdef __BMI2__
|
||||
-#include <bmi2intrin.h>
|
||||
+#include <immintrin.h>
|
||||
#endif // __BMI2__
|
||||
|
||||
namespace absl {
|
||||
@@ -0,0 +1,38 @@
|
||||
# Abseil accepted the upstream fix: https://github.com/abseil/abseil-cpp/pull/2071
|
||||
# Keep this workaround until Electron, Deno's rusty_v8, and Signal's WebRTC
|
||||
# update their bundled Abseil copies to include it.
|
||||
{ prev }:
|
||||
let
|
||||
patchAbseilBmi2Include =
|
||||
package:
|
||||
package.overrideAttrs (old: {
|
||||
# GCC and Clang require the public umbrella header for BMI2 intrinsics.
|
||||
patches = (old.patches or [ ]) ++ [ ./bmi2-public-header.patch ];
|
||||
});
|
||||
|
||||
electron43Unwrapped = patchAbseilBmi2Include prev.electron_43.unwrapped;
|
||||
electron43 = prev.electron_43.override {
|
||||
electron-unwrapped = electron43Unwrapped;
|
||||
};
|
||||
|
||||
signalCallPackage =
|
||||
path: args:
|
||||
let
|
||||
package = prev.callPackage path args;
|
||||
in
|
||||
if builtins.baseNameOf path == "webrtc.nix" then patchAbseilBmi2Include package else package;
|
||||
in
|
||||
prev.lib.optionalAttrs ((prev.stdenv.hostPlatform.gcc.arch or null) == "x86-64-v3") {
|
||||
deno =
|
||||
let
|
||||
librusty_v8 = patchAbseilBmi2Include prev.deno.passthru.librusty_v8;
|
||||
in
|
||||
prev.deno.override { inherit librusty_v8; };
|
||||
|
||||
electron_43 = electron43;
|
||||
|
||||
signal-desktop = prev.signal-desktop.override {
|
||||
electron_43 = electron43;
|
||||
callPackage = signalCallPackage;
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
_final: prev:
|
||||
(import ./abseil { inherit prev; })
|
||||
// {
|
||||
gnutls = import ./gnutls { inherit (prev) gnutls; };
|
||||
prometheus = import ./prometheus { inherit (prev) prometheus; };
|
||||
|
||||
pythonPackagesExtensions = prev.pythonPackagesExtensions ++ [
|
||||
(_pythonFinal: pythonPrev: {
|
||||
pytest-xdist = import ./pytest-xdist { inherit (pythonPrev) pytest-xdist; };
|
||||
scipy = import ./scipy { inherit (pythonPrev) scipy; };
|
||||
sentry-sdk = import ./sentry-sdk { inherit (pythonPrev) sentry-sdk; };
|
||||
})
|
||||
];
|
||||
}
|
||||
@@ -0,0 +1,160 @@
|
||||
# GnuTLS UDP server readiness
|
||||
|
||||
Under load, the test client can start before `gnutls-serv` binds its UDP
|
||||
socket, and the first handshake fails with `Connection refused`.
|
||||
`serv-udp.sh` currently waits a fixed four seconds; elapsed time does not
|
||||
establish server readiness. `udp-server-readiness.patch` replaces that wait
|
||||
with polling for the local IPv4 UDP endpoint.
|
||||
|
||||
## Scope and waiting behavior
|
||||
|
||||
The patch changes the existing `wait_udp_server()` and adds a new
|
||||
`check_if_udp_port_bound()` beside it in `tests/scripts/common.sh`.
|
||||
`serv-udp.sh` is its only caller in 3.8.13. The TCP helpers `wait_server()`
|
||||
and `wait_for_port()`, including their existing sleeps, are unchanged.
|
||||
Both original DTLS handshake checks remain unchanged.
|
||||
|
||||
Each iteration checks process liveness and the socket **before sleeping**.
|
||||
A ready socket returns immediately. An unsuccessful check sleeps two
|
||||
seconds only if another attempt remains: at most 90 attempts, consistent
|
||||
with the existing `wait_server()` budget implemented by `wait_for_port()`,
|
||||
with no sleep after the final check. Server exit fails early; exhausting the
|
||||
budget fails and terminates the server. No handshake is retried, and no
|
||||
protocol timeout is changed. Once bound, the kernel can queue datagrams
|
||||
while the server is scheduled; the probe itself sends no packets.
|
||||
|
||||
The existing `have_port_finder()` prefers `ss`, then `netstat`. If neither
|
||||
exists, it prints `neither ss nor netstat found` and exits **77 (skip)**.
|
||||
In the normal test flow, port selection calls it before launching a server.
|
||||
The probe runs in a subshell so that, even if this skip occurs after launch,
|
||||
the waiting helper can terminate and reap the server before exiting 77.
|
||||
|
||||
## Why an IPv4 socket is expected
|
||||
|
||||
This is specific to the server used by this test, not a general rule that
|
||||
IPv6 sockets cannot serve IPv4 clients. The client explicitly uses
|
||||
`127.0.0.1`. The server's `--udp` path calls `udp_server()`, which calls
|
||||
`listen_socket(..., SOCK_DGRAM)`. That function iterates the wildcard
|
||||
addresses returned by `getaddrinfo(NULL, ..., AI_PASSIVE)`:
|
||||
|
||||
| Server build / Linux setting | Binding behavior |
|
||||
| --- | --- |
|
||||
| IPv6 enabled, `net.ipv6.bindv6only=0` | Requests `IPV6_V6ONLY=1` on the IPv6 socket, binds `[::]:PORT`, and separately binds `0.0.0.0:PORT`. It overrides the system's dual-stack default. |
|
||||
| IPv6 enabled, `net.ipv6.bindv6only=1` | The same explicit socket option and separate IPv4/IPv6 binds. |
|
||||
| `HAVE_IPV6` undefined | Skips every address family except `AF_INET`; only the IPv4 wildcard is attempted. |
|
||||
|
||||
`udp_server()` uses `wait_for_connection()`, which puts **every listener**
|
||||
from that list into `select()` and returns a readable socket for `recvfrom()`;
|
||||
it does not permanently choose one socket based on `getaddrinfo()` order.
|
||||
|
||||
The first two cases were traced with the actual GnuTLS 3.8.13 binary in
|
||||
separate Linux network namespaces: `setsockopt(IPV6_V6ONLY, [1])` and both
|
||||
UDP binds returned success under each setting. The no-IPv6 case was checked
|
||||
in source, not by building a second binary. The same bind implementation
|
||||
was checked directly on GitLab master.
|
||||
|
||||
Thus, successful normal startup for this invocation provides an explicit
|
||||
IPv4 socket; a lone IPv6 wildcard is not the expected success path.
|
||||
There is one portability caveat: upstream discards the return value of
|
||||
`setsockopt(IPV6_V6ONLY)`. On a platform where that call fails and the server
|
||||
ends up with only a dual-stack socket, this helper would time out despite
|
||||
IPv4 reachability. Such a platform needs additional handling before this
|
||||
patch can claim support. Blindly accepting every IPv6 wildcard would also
|
||||
accept IPv6-only sockets before the separate IPv4 bind finishes.
|
||||
|
||||
Source: [`src/serv.c`, `listen_socket()`](https://gitlab.com/gnutls/gnutls/-/blob/master/src/serv.c#L937),
|
||||
[`src/udp-serv.c`](https://gitlab.com/gnutls/gnutls/-/blob/master/src/udp-serv.c),
|
||||
and [`tests/serv-udp.sh`](https://gitlab.com/gnutls/gnutls/-/blob/master/tests/serv-udp.sh).
|
||||
|
||||
## Port matching and ownership limit
|
||||
|
||||
Only `-an` is passed to the socket-listing tool: BSD `netstat -u` selects
|
||||
Unix-domain sockets, whereas Linux `netstat -u` selects UDP. The parser
|
||||
handles the extra state column in `ss`, Linux colon-separated endpoints,
|
||||
and BSD dot-separated endpoints, including `*.PORT`. It matches the full
|
||||
local port and rejects TCP, IPv6 entries, peer ports, and longer numbers.
|
||||
|
||||
A live PID plus a bound port does **not** prove that PID owns the socket.
|
||||
Existing `GETPORT` selection checks for an unused port and uses a test
|
||||
port-lock directory; `launch_bare_server()` also calls
|
||||
`wait_for_free_port()` before starting the process. These are advisory:
|
||||
the launcher does not enforce the latter's result, and another process
|
||||
can bind between the check and launch. The patch does not close that race
|
||||
or add nonportable PID parsing. An unrelated process can satisfy the
|
||||
socket check; the real handshakes remain the functional check and may
|
||||
fail (or reach the wrong server). This is a startup-order fix, not a
|
||||
socket-ownership guarantee.
|
||||
|
||||
## Reproduction and focused checks
|
||||
|
||||
Apply the patch to an unpacked source tree, then run the companion checks
|
||||
with Python's standard library and a shell:
|
||||
|
||||
```sh
|
||||
patch --fuzz=0 -d /path/to/gnutls -p1 < udp-server-readiness.patch
|
||||
SHELL=/bin/sh python3 verify-readiness.py /path/to/gnutls/tests/scripts/common.sh -v
|
||||
```
|
||||
|
||||
Set `NETSTAT=/path/to/netstat` to exercise one outside `PATH`. The checks
|
||||
cover Linux/BSD output samples, false matches, immediate readiness,
|
||||
missing tools, process exit, timeout cleanup, and real IPv4 UDP sockets
|
||||
whose bind is delayed six seconds. The missing-tools fixture is skipped
|
||||
if an absolute fallback `ss` path cannot be hidden with `PATH`. Native
|
||||
BSD execution remains untested.
|
||||
|
||||
To reproduce with GnuTLS itself, run `tests/serv-udp.sh` with `SERV` pointing
|
||||
to a wrapper that sleeps six seconds, then `exec`s `gnutls-serv` with all
|
||||
arguments. Set `CLI` to the matching `gnutls-cli`, `srcdir` to the source
|
||||
`tests` directory, and `abs_top_builddir` to a writable build directory.
|
||||
With GnuTLS 3.8.13, the original helper failed the first handshake with
|
||||
`Connection refused`; the patched helper passed both with the same binaries.
|
||||
|
||||
## GnuTLS submission
|
||||
|
||||
Development and merge requests are on [GitLab](https://gitlab.com/gnutls/gnutls).
|
||||
[`CONTRIBUTING.md` on master](https://gitlab.com/gnutls/gnutls/-/blob/master/CONTRIBUTING.md)
|
||||
was read directly for this review. It requires the contributor's DCO
|
||||
`Signed-off-by`, successful and failure test coverage, consistent coding
|
||||
style, and adequate documentation; GitLab CI runs for merge requests.
|
||||
Its commenting guidance asks for comments explaining non-obvious behavior
|
||||
or protocol expectations. It does not prescribe an additional special
|
||||
test-suite comment. The patch now explains its IPv4 binding assumption
|
||||
next to the probe.
|
||||
|
||||
The submission will contain the shell patch, without the Python verifier
|
||||
or a new Python test dependency. The existing `serv-udp.sh` supplies the
|
||||
functional success check. Running it through the six-second startup
|
||||
wrapper supplies a reproducible regression case: it fails before the fix
|
||||
and passes after it. The local verifier was used to validate socket-output
|
||||
parsing and the helper's success, process-exit, skip-cleanup, and timeout
|
||||
branches. Those branch checks are local evidence, not new automated
|
||||
coverage in the upstream suite; the MR must state that distinction.
|
||||
|
||||
No dedicated unit-test harness for these shell helpers was found in the
|
||||
3.8.13 tests inspected. That does not establish that Python cannot be used
|
||||
upstream; keeping this submission dependency-free is a scope choice. Use
|
||||
the existing test and before/after reproduction as the submission's
|
||||
coverage argument, retaining the platform limitations above. Apply the
|
||||
patch in an upstream checkout and include those results with the
|
||||
contributor's own sign-off. No MR or sign-off has been created.
|
||||
|
||||
## Local NixOS integration and build results
|
||||
|
||||
`overlays/default.nix` imports the `overlays/patches` overlay, which loads
|
||||
`gnutls/default.nix` to apply the patch and keep `serv-udp.sh` enabled.
|
||||
The patch itself has no Nix dependencies and applies to 3.8.13 and GitLab
|
||||
master without fuzz.
|
||||
|
||||
The final patch was rebuilt with:
|
||||
|
||||
```sh
|
||||
nix build --no-link -L .#nixosConfigurations.jeeves.pkgs.gnutls
|
||||
```
|
||||
|
||||
That x86-64-v3 build passed: 927 tests, 796 passes, 131 existing skips,
|
||||
zero failures/errors, and `PASS: serv-udp.sh`. The patch bytes in the built
|
||||
derivation were compared with the repository artifact; both have SHA-256
|
||||
`59013d47fd446f2dd065012a2259ccc1898fedc8a053a630e13efa0076368760`.
|
||||
All seven local checks passed, including skip cleanup and exactly 90
|
||||
probes with 89 sleeps on timeout. The six-second before/after reproduction
|
||||
was also repeated successfully with the final helper.
|
||||
@@ -0,0 +1,6 @@
|
||||
{ gnutls }:
|
||||
gnutls.overrideAttrs (old: {
|
||||
# Keep the UDP handshake test enabled on loaded builders by waiting for
|
||||
# the server to bind its socket. Kept as a standalone patch for upstream.
|
||||
patches = (old.patches or [ ]) ++ [ ./udp-server-readiness.patch ];
|
||||
})
|
||||
@@ -0,0 +1,70 @@
|
||||
Subject: [PATCH] tests: wait for the UDP server socket before connecting
|
||||
|
||||
A fixed four-second sleep does not guarantee that gnutls-serv has bound
|
||||
its UDP socket on a busy builder. Poll the local IPv4 UDP endpoint using
|
||||
the existing ss/netstat discovery, with the same retry budget as the TCP
|
||||
helper. Fail early if the server exits, and retain the original handshake
|
||||
checks in serv-udp.sh.
|
||||
|
||||
Use flags common to ss and BSD/Linux netstat. Match the local endpoint
|
||||
and complete port number, excluding TCP, IPv6-only and peer endpoints.
|
||||
|
||||
--- a/tests/scripts/common.sh
|
||||
+++ b/tests/scripts/common.sh
|
||||
@@ -185,10 +185,55 @@
|
||||
fi
|
||||
}
|
||||
|
||||
+check_if_udp_port_bound() {
|
||||
+ local PORT=$1
|
||||
+ have_port_finder
|
||||
+ # Use only -an, which is shared by ss and BSD/Linux netstat. UDP has
|
||||
+ # no LISTEN state. Match the local IPv4 endpoint, not a peer port or
|
||||
+ # a longer port number. serv-udp.sh connects to 127.0.0.1;
|
||||
+ # listen_socket() in serv.c binds IPv4 separately and requests
|
||||
+ # IPV6_V6ONLY=1 for its IPv6 socket.
|
||||
+ $PFCMD -an | awk -v port="$PORT" '
|
||||
+ $1 == "udp" || $1 == "udp4" {
|
||||
+ # ss includes a state column; netstat does not.
|
||||
+ address = ($2 == "UNCONN" || $2 == "ESTAB") ? $5 : $4
|
||||
+ if (address ~ ("^[0-9.]+[.:]" port "$") ||
|
||||
+ address == "*." port)
|
||||
+ found = 1
|
||||
+ }
|
||||
+ END { exit !found }
|
||||
+ '
|
||||
+}
|
||||
+
|
||||
wait_udp_server() {
|
||||
local PID=$1
|
||||
+ local ret
|
||||
trap "test -n \"${PID}\" && kill ${PID};exit 1" 1 15 2
|
||||
- sleep 4
|
||||
+ local i=0
|
||||
+ # Use the same retry budget as wait_for_port(), but also stop if the
|
||||
+ # server exits before binding its socket.
|
||||
+ while test $i -lt 90; do
|
||||
+ if ! kill -0 "$PID" 2>/dev/null; then
|
||||
+ fail "" "UDP server $PID exited before binding port $PORT"
|
||||
+ fi
|
||||
+ # Contain have_port_finder's exit so a skip also stops the server.
|
||||
+ if (check_if_udp_port_bound "$PORT"); then
|
||||
+ return 0
|
||||
+ else
|
||||
+ ret=$?
|
||||
+ if test "$ret" = 77; then
|
||||
+ kill "$PID" 2>/dev/null || :
|
||||
+ wait "$PID" 2>/dev/null || :
|
||||
+ exit 77
|
||||
+ fi
|
||||
+ fi
|
||||
+ i=$((i + 1))
|
||||
+ if test $i -lt 90; then
|
||||
+ echo "try $i: waiting for UDP port $PORT"
|
||||
+ sleep 2
|
||||
+ fi
|
||||
+ done
|
||||
+ fail "$PID" "UDP server $PORT did not come up"
|
||||
}
|
||||
|
||||
create_testdir() {
|
||||
Executable
+180
@@ -0,0 +1,180 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Exercise patched common.sh without building GnuTLS (Python standard library only).
|
||||
|
||||
Usage: python3 verify-readiness.py /path/to/patched/tests/scripts/common.sh
|
||||
Set SHELL to test another shell, and NETSTAT to test a netstat outside PATH.
|
||||
"""
|
||||
|
||||
# Use unittest so this upstream companion tool needs no pytest installation.
|
||||
# ruff: noqa: PT009
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
COMMON = str(Path(sys.argv.pop(1)).resolve())
|
||||
SHELL = os.environ.get("SHELL", "/bin/sh")
|
||||
|
||||
|
||||
class ReadinessTests(unittest.TestCase):
|
||||
"""Check endpoint parsing and the server startup lifecycle."""
|
||||
|
||||
def setUp(self) -> None:
|
||||
"""Create a socket-listing fixture for each check."""
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.tmp.cleanup)
|
||||
self.root = Path(self.tmp.name)
|
||||
self.fixture = self.root / "sockets"
|
||||
self.fixture.write_text("")
|
||||
self.finder = self.root / "port-finder"
|
||||
self.finder.write_text('#!/bin/sh\ncat "$SOCKET_FIXTURE"\n')
|
||||
self.finder.chmod(0o755)
|
||||
|
||||
def run_shell(self, body: str, **env: str) -> subprocess.CompletedProcess[str]:
|
||||
"""Source the actual helper and run a shell scenario."""
|
||||
return subprocess.run(
|
||||
[SHELL, "-c", '. "$COMMON"\n' + body],
|
||||
env={
|
||||
**os.environ,
|
||||
"COMMON": COMMON,
|
||||
"SOCKET_FIXTURE": str(self.fixture),
|
||||
"PFCMD": str(self.finder),
|
||||
"PORT": "12345",
|
||||
**env,
|
||||
},
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=20,
|
||||
check=False,
|
||||
)
|
||||
|
||||
def test_socket_formats_and_false_matches(self) -> None:
|
||||
"""Accept IPv4 UDP local endpoints and reject unrelated sockets."""
|
||||
cases = [
|
||||
("udp UNCONN 0 0 0.0.0.0:12345 0.0.0.0:*", True),
|
||||
("udp UNCONN 0 0 127.0.0.1:12345 0.0.0.0:*", True),
|
||||
("udp 0 0 0.0.0.0:12345 0.0.0.0:*", True),
|
||||
("udp4 0 0 *.12345 *.*", True),
|
||||
("udp 0 0 127.0.0.1.12345 *.*", True),
|
||||
("udp 0 0 *.12345 *.*", True),
|
||||
("udp UNCONN 0 0 0.0.0.0:123456 0.0.0.0:*", False),
|
||||
("udp 0 0 0.0.0.0:123456 0.0.0.0:*", False),
|
||||
("udp ESTAB 0 0 127.0.0.1:54321 127.0.0.1:12345", False),
|
||||
("udp 0 0 127.0.0.1:54321 127.0.0.1:12345", False),
|
||||
("tcp LISTEN 0 128 0.0.0.0:12345 0.0.0.0:*", False),
|
||||
("tcp 0 0 0.0.0.0:12345 0.0.0.0:* LISTEN", False),
|
||||
("udp UNCONN 0 0 [::]:12345 [::]:*", False),
|
||||
("udp UNCONN 0 0 *:12345 *:*", False),
|
||||
("udp6 0 0 :::12345 :::*", False),
|
||||
("udp6 0 0 *.12345 *.*", False),
|
||||
("", False),
|
||||
]
|
||||
for row, ready in cases:
|
||||
with self.subTest(row=row):
|
||||
self.fixture.write_text(row + "\n")
|
||||
result = self.run_shell('check_if_udp_port_bound "$PORT"')
|
||||
self.assertEqual(result.returncode, 0 if ready else 1, result.stderr)
|
||||
|
||||
def test_exited_server_fails_immediately(self) -> None:
|
||||
"""Fail without sleeping when the server has already exited."""
|
||||
result = self.run_shell(
|
||||
'true &\npid=$!\nwait "$pid"\nsleep() { echo "unexpected sleep" >&2; }\nwait_udp_server "$pid"'
|
||||
)
|
||||
self.assertEqual(result.returncode, 1)
|
||||
self.assertIn("exited before binding", result.stderr)
|
||||
self.assertNotIn("unexpected sleep", result.stderr)
|
||||
|
||||
def test_ready_socket_does_not_sleep(self) -> None:
|
||||
"""Check readiness before the first sleep."""
|
||||
self.fixture.write_text("udp UNCONN 0 0 0.0.0.0:12345 0.0.0.0:*\n")
|
||||
result = self.run_shell('sleep() { echo "unexpected sleep" >&2; }\nwait_udp_server "$$"')
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertNotIn("unexpected sleep", result.stderr)
|
||||
|
||||
def test_missing_port_finders_skip(self) -> None:
|
||||
"""Skip and stop the live server when no finder is available."""
|
||||
# have_port_finder also tries these paths independently of PATH.
|
||||
if any(os.access(f"{directory}/ss", os.X_OK) for directory in ("/sbin", "/usr/sbin", "/usr/local/sbin")):
|
||||
self.skipTest("an absolute ss path cannot be hidden by this PATH-only fixture")
|
||||
with subprocess.Popen(["sleep", "60"]) as server:
|
||||
try:
|
||||
result = self.run_shell(
|
||||
'unset PFCMD\nPATH=/nonexistent\nwait_udp_server "$SERVER_PID"',
|
||||
SERVER_PID=str(server.pid),
|
||||
)
|
||||
self.assertEqual(result.returncode, 77)
|
||||
self.assertIn("neither ss nor netstat found", result.stderr)
|
||||
server.wait(timeout=3)
|
||||
self.assertLess(server.returncode, 0)
|
||||
finally:
|
||||
if server.poll() is None:
|
||||
server.kill()
|
||||
|
||||
def test_timeout_is_bounded_and_cleans_up(self) -> None:
|
||||
"""Stop polling after the retry budget and terminate the server."""
|
||||
# Only accelerate the polling delay; keep a real live server process.
|
||||
self.finder.write_text('#!/bin/sh\necho probe >&2\ncat "$SOCKET_FIXTURE"\n')
|
||||
with subprocess.Popen(["sleep", "60"]) as server:
|
||||
try:
|
||||
result = self.run_shell(
|
||||
'sleep() { echo polling-sleep; }\nwait_udp_server "$SERVER_PID"',
|
||||
SERVER_PID=str(server.pid),
|
||||
)
|
||||
self.assertEqual(result.returncode, 1)
|
||||
self.assertIn("did not come up", result.stderr)
|
||||
self.assertEqual(result.stderr.count("probe\n"), 90)
|
||||
self.assertEqual(result.stdout.count("polling-sleep"), 89)
|
||||
server.wait(timeout=3)
|
||||
self.assertLess(server.returncode, 0)
|
||||
finally:
|
||||
if server.poll() is None:
|
||||
server.kill()
|
||||
|
||||
def test_server_exits_while_waiting(self) -> None:
|
||||
"""Detect a startup failure that happens after polling begins."""
|
||||
result = self.run_shell('sleep 1 &\npid=$!\nwait_udp_server "$pid"')
|
||||
self.assertEqual(result.returncode, 1)
|
||||
self.assertIn("exited before binding", result.stderr)
|
||||
self.assertIn("waiting for UDP port", result.stdout)
|
||||
|
||||
def test_real_socket_delayed_beyond_four_seconds(self) -> None:
|
||||
"""Wait for a real delayed bind with each installed port finder."""
|
||||
finders = [shutil.which("ss"), os.environ.get("NETSTAT") or shutil.which("netstat")]
|
||||
finders = [finder for finder in finders if finder]
|
||||
if not finders:
|
||||
self.skipTest("neither ss nor netstat available")
|
||||
for finder in finders:
|
||||
with self.subTest(finder=finder):
|
||||
with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as sock:
|
||||
sock.bind(("127.0.0.1", 0))
|
||||
port = sock.getsockname()[1]
|
||||
code = (
|
||||
"import socket,time,sys; time.sleep(6); "
|
||||
"s=socket.socket(socket.AF_INET,socket.SOCK_DGRAM); "
|
||||
"s.bind(('127.0.0.1',int(sys.argv[1]))); time.sleep(30)"
|
||||
)
|
||||
with subprocess.Popen([sys.executable, "-c", code, str(port)]) as server:
|
||||
try:
|
||||
started = time.monotonic()
|
||||
result = self.run_shell(
|
||||
'wait_udp_server "$SERVER_PID"',
|
||||
SERVER_PID=str(server.pid),
|
||||
PORT=str(port),
|
||||
PFCMD=finder,
|
||||
)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertGreaterEqual(time.monotonic() - started, 6)
|
||||
self.assertIsNone(server.poll())
|
||||
finally:
|
||||
server.terminate()
|
||||
server.wait(timeout=3)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,66 @@
|
||||
# Prometheus complete test parsing
|
||||
|
||||
CodeMirror gives editor-state creation a 20 ms synchronous parsing budget.
|
||||
The shared `createEditorState()` test helper can therefore return an
|
||||
incomplete syntax tree when the process is descheduled. The completion and
|
||||
vector-matching tests immediately inspect that tree.
|
||||
|
||||
## Scope and behavior
|
||||
|
||||
`complete-test-parsing.patch` changes only
|
||||
`module/codemirror-promql/src/test/utils-test.ts` inside `web/ui`. It completes
|
||||
the small test expression with `ensureSyntaxTree(..., Infinity)` and publishes
|
||||
the completed parse through an empty transaction so `syntaxTree(state)` sees
|
||||
it. Failure to obtain a tree raises an error.
|
||||
|
||||
The original assertions remain enabled, including `autocomplete topk params 2`
|
||||
and `foo * on(test,blub) bar`. The unlimited budget applies to the test helper;
|
||||
production editor parsing budgets are unchanged.
|
||||
|
||||
## Reproduction and focused checks
|
||||
|
||||
Use a disposable Prometheus 3.14.0 checkout. The patch root is `web/ui`, matching
|
||||
the Nix assets derivation. From this directory:
|
||||
|
||||
```sh
|
||||
patch --fuzz=0 -d /path/to/prometheus/web/ui -p1 < complete-test-parsing.patch
|
||||
cd /path/to/prometheus/web/ui
|
||||
pnpm install --frozen-lockfile
|
||||
pnpm --filter @prometheus-io/lezer-promql build
|
||||
pnpm --filter @prometheus-io/codemirror-promql test
|
||||
```
|
||||
|
||||
To force the scheduling condition, temporarily append this clock to
|
||||
`module/codemirror-promql/setupJest.cjs` in the disposable checkout:
|
||||
|
||||
```js
|
||||
let parseClock = 0;
|
||||
Date.now = () => (parseClock += 25);
|
||||
```
|
||||
|
||||
Each clock read crosses the editor's initial parsing budget. Against the
|
||||
original helper, the hybrid and vector suites have 186 failures, including
|
||||
both locally excluded cases. With the patch, all 386 CodeMirror tests pass
|
||||
under that same clock. Remove the injected clock before normal builds.
|
||||
|
||||
## Upstream status
|
||||
|
||||
This is a standalone test-helper patch for Prometheus 3.14.0. No upstream
|
||||
submission was made during this work. Recheck the helper when updating
|
||||
Prometheus or CodeMirror, including how an ensured parse becomes visible
|
||||
through the editor state.
|
||||
|
||||
## Local NixOS integration and build results
|
||||
|
||||
[`../default.nix`](../default.nix) loads `default.nix`, which patches the
|
||||
separate assets derivation. It updates both `passthru.assets` and the main
|
||||
Prometheus build's reference to those assets. From the repository root:
|
||||
|
||||
```sh
|
||||
nix build --no-link -L .#nixosConfigurations.jeeves.pkgs.prometheus.assets
|
||||
```
|
||||
|
||||
The full x86-64-v3 assets build passed with the normal clock, including the
|
||||
CodeMirror and UI suites. Host-flake evaluation confirmed that the main
|
||||
Prometheus derivation refers to these patched assets. The Go server package
|
||||
was not rebuilt for this test-helper change.
|
||||
@@ -0,0 +1,36 @@
|
||||
Subject: [PATCH] tests: finish parsing before inspecting editor state
|
||||
|
||||
EditorState creation has a 20 ms parsing budget. A descheduled test can
|
||||
therefore observe an incomplete tree. Finish these small test documents
|
||||
without an interactive deadline and publish the result with a transaction.
|
||||
Keep the original completion and vector-matching assertions enabled.
|
||||
|
||||
--- a/module/codemirror-promql/src/test/utils-test.ts
|
||||
+++ b/module/codemirror-promql/src/test/utils-test.ts
|
||||
@@ -13,7 +13,7 @@
|
||||
|
||||
import { parser } from '@prometheus-io/lezer-promql';
|
||||
import { EditorState } from '@codemirror/state';
|
||||
-import { LRLanguage } from '@codemirror/language';
|
||||
+import { ensureSyntaxTree, LRLanguage } from '@codemirror/language';
|
||||
import nock from 'nock';
|
||||
import path from 'path';
|
||||
import { fileURLToPath } from 'url';
|
||||
@@ -23,10 +23,16 @@
|
||||
const __dirname = path.dirname(fileURLToPath(import.meta.url));
|
||||
|
||||
export function createEditorState(expr: string): EditorState {
|
||||
- return EditorState.create({
|
||||
+ const state = EditorState.create({
|
||||
doc: expr,
|
||||
extensions: lightPromQLSyntax,
|
||||
});
|
||||
+ // These tests need a complete tree, independent of the editor's time budget.
|
||||
+ if (!ensureSyntaxTree(state, state.doc.length, Infinity)) {
|
||||
+ throw new Error('Unable to parse the test expression');
|
||||
+ }
|
||||
+ // Publish the completed parse so syntaxTree(state) sees it too.
|
||||
+ return state.update({}).state;
|
||||
}
|
||||
|
||||
export function mockPrometheusServer(): void {
|
||||
@@ -0,0 +1,17 @@
|
||||
{ prometheus }:
|
||||
prometheus.overrideAttrs (
|
||||
old:
|
||||
let
|
||||
assets = old.passthru.assets.overrideAttrs (assetsOld: {
|
||||
patches = (assetsOld.patches or [ ]) ++ [ ./complete-test-parsing.patch ];
|
||||
});
|
||||
in
|
||||
{
|
||||
postPatch = builtins.replaceStrings [ "${old.passthru.assets}" ] [ "${assets}" ] (
|
||||
builtins.unsafeDiscardStringContext old.postPatch
|
||||
);
|
||||
passthru = old.passthru // {
|
||||
inherit assets;
|
||||
};
|
||||
}
|
||||
)
|
||||
@@ -0,0 +1,79 @@
|
||||
# pytest-xdist test fixes
|
||||
|
||||
With two workers and a restart limit of three, the fourth worker crash
|
||||
requests shutdown while another test can still be running. That test may
|
||||
also crash. The original queued-work test requires exactly four failures,
|
||||
even though five failures can occur without exceeding the replacement limit.
|
||||
|
||||
## Scope and behavior
|
||||
|
||||
`concurrent-worker-crashes.patch` changes the assertions in
|
||||
`TestNodeFailure.test_max_worker_restart_tests_queued` in
|
||||
`testing/acceptance_test.py`. It requires exactly three replacements, four or
|
||||
five failed tests, the failed-tests exit status, the limit message, and no
|
||||
internal error. It retains the two-worker workload and ten queued tests.
|
||||
|
||||
`worker-startup-timeout.patch` changes the remote-test helper's event timeout
|
||||
from 10 to 60 seconds so loaded builders have time to start workers. The
|
||||
helper returns immediately when an event arrives and still has a bounded wait.
|
||||
|
||||
The existing nixpkgs pytest-9 compatibility patches remain in place.
|
||||
Production scheduling and worker-restart behavior are unchanged.
|
||||
|
||||
## Reproduction and focused checks
|
||||
|
||||
Use a disposable pytest-xdist 3.8.0 checkout with its test dependencies and
|
||||
the nixpkgs pytest-9 compatibility patches where required. From this directory:
|
||||
|
||||
```sh
|
||||
patch --fuzz=0 -d /path/to/pytest-xdist -p1 < concurrent-worker-crashes.patch
|
||||
patch --fuzz=0 -d /path/to/pytest-xdist -p1 < worker-startup-timeout.patch
|
||||
cd /path/to/pytest-xdist
|
||||
python -m pytest testing/acceptance_test.py \
|
||||
-k test_max_worker_restart_tests_queued -q
|
||||
python -m pytest testing/test_remote.py -q
|
||||
```
|
||||
|
||||
Twenty unmodified runs passed during the review. To force the failing
|
||||
schedule, modify the generated crashing test in a disposable checkout to
|
||||
accept `worker_id`: make `gw3` wait for a marker created by `gw4`, and make
|
||||
`gw4` pause 0.1 seconds after creating the marker. Then both have in-flight
|
||||
tests when shutdown starts. Bound the marker wait so a reproduction failure
|
||||
cannot hang the suite. The original assertion fails on five reported
|
||||
failures; the patched test passes.
|
||||
|
||||
## Worker startup and outer concurrency
|
||||
|
||||
[`default.nix`](default.nix) runs the outer suite with one worker to limit
|
||||
nested process pools. This is a Nix test-runner setting; the source timeout
|
||||
change lives in [`worker-startup-timeout.patch`](worker-startup-timeout.patch).
|
||||
|
||||
A separate reproduction inserts an 11-second `pytest_sessionstart` delay
|
||||
into the child created by `test_basic_collect_and_runtests` in
|
||||
`testing/test_remote.py`. The original 10-second channel wait fails; the
|
||||
60-second wait passes. This bounds waits for test worker events, including
|
||||
startup, rather than changing a product deadline.
|
||||
|
||||
## Upstream status
|
||||
|
||||
These are standalone test patches for pytest-xdist 3.8.0. No upstream submission
|
||||
was made during this work. Recheck the allowed in-flight failures, replacement
|
||||
count, and remote-test wait when updating the scheduler or worker behavior.
|
||||
|
||||
## Local NixOS integration and build results
|
||||
|
||||
[`../default.nix`](../default.nix) loads `default.nix` through
|
||||
`pythonPackagesExtensions`. From the repository root:
|
||||
|
||||
```sh
|
||||
nix build --no-link -L .#nixosConfigurations.jeeves.pkgs.python314Packages.pytest-xdist
|
||||
```
|
||||
|
||||
After consolidating the settings in this directory, the full x86-64-v3 package
|
||||
build passed 185 tests, with 6 existing skips and 10 expected failures. Nix
|
||||
evaluation confirmed the same outer-worker limit and preserved existing
|
||||
patches, with the timeout now applied as a source patch.
|
||||
|
||||
The earlier forced concurrent-crash and delayed-startup reproductions passed
|
||||
after their fixes; the focused crash test also passed after formatting its
|
||||
assertion.
|
||||
@@ -0,0 +1,29 @@
|
||||
Subject: [PATCH] tests: count replacements when checking the worker restart limit
|
||||
|
||||
With two workers, another in-flight test may crash after the fourth
|
||||
crash requests shutdown. Either four or five failed tests is valid.
|
||||
Require exactly three replacements and the failed-tests exit status,
|
||||
while preserving the queued-work and no-internal-error assertions.
|
||||
|
||||
--- a/testing/acceptance_test.py
|
||||
+++ b/testing/acceptance_test.py
|
||||
@@ -1011,9 +1011,18 @@
|
||||
"worker*crashed while running*",
|
||||
"worker*crashed while running*",
|
||||
"* xdist: maximum crashed workers reached: 3 *",
|
||||
- "* 4 failed in *",
|
||||
]
|
||||
)
|
||||
+ # A second in-flight test may crash after shutdown is requested.
|
||||
+ # The restart limit constrains replacements, not concurrent failures.
|
||||
+ replacements = sum(
|
||||
+ line.startswith("replacing crashed worker ") for line in res.stdout.lines
|
||||
+ )
|
||||
+ assert replacements == 3
|
||||
+ failed = res.parseoutcomes()["failed"]
|
||||
+ assert failed in (4, 5)
|
||||
+ res.assert_outcomes(failed=failed)
|
||||
+ assert res.ret == pytest.ExitCode.TESTS_FAILED
|
||||
assert "INTERNALERROR" not in res.stdout.str()
|
||||
|
||||
def test_max_worker_restart_die(self, pytester: pytest.Pytester) -> None:
|
||||
@@ -0,0 +1,12 @@
|
||||
{ pytest-xdist }:
|
||||
pytest-xdist.overridePythonAttrs (old: {
|
||||
patches = (old.patches or [ ]) ++ [
|
||||
./concurrent-worker-crashes.patch
|
||||
./worker-startup-timeout.patch
|
||||
];
|
||||
|
||||
# The suite exercises its own worker pools. Limit the outer suite to one worker.
|
||||
preCheck = builtins.replaceStrings [ "--numprocesses=$NIX_BUILD_CORES" ] [ "--numprocesses=1" ] (
|
||||
old.preCheck or ""
|
||||
);
|
||||
})
|
||||
@@ -0,0 +1,19 @@
|
||||
Subject: [PATCH] tests: allow more time for remote worker events
|
||||
|
||||
Worker startup can exceed ten seconds on heavily loaded builders. Allow
|
||||
the remote-test helper to wait up to sixty seconds for worker events.
|
||||
The wait still returns as soon as an event arrives and remains bounded.
|
||||
Production worker timeouts and test assertions are unchanged.
|
||||
|
||||
--- a/testing/test_remote.py
|
||||
+++ b/testing/test_remote.py
|
||||
@@ -17,7 +17,8 @@
|
||||
from xdist.workermanage import WorkerController
|
||||
|
||||
|
||||
-WAIT_TIMEOUT = 10.0
|
||||
+# Allow worker events extra time on heavily loaded builders.
|
||||
+WAIT_TIMEOUT = 60.0
|
||||
|
||||
|
||||
def check_marshallable(d: object) -> None:
|
||||
@@ -0,0 +1,70 @@
|
||||
# SciPy STFT test tolerances
|
||||
|
||||
The x86-64-v3 build can produce small floating-point residuals in inverse-STFT
|
||||
comparisons and scaling round trips. The original bounds reject these results,
|
||||
including residuals around `4e-17` where a round trip expects zero for a signal
|
||||
with amplitude 2.
|
||||
|
||||
## Scope and behavior
|
||||
|
||||
`stft-test-tolerances.patch` changes only the signal tests:
|
||||
|
||||
- The inverse-STFT comparison in `_scipy_spectral_test_shim.py` uses
|
||||
`max(1e-7, 2 * np.finfo(x.dtype).eps)` as its relative tolerance. Float64
|
||||
keeps the original bound, and the existing i686 override remains.
|
||||
- Three scaling round trips in `test_spectral.py` gain an absolute tolerance
|
||||
of one epsilon for the input dtype, allowing small residuals near zero.
|
||||
|
||||
The tests remain enabled, and the production STFT implementation is unchanged.
|
||||
|
||||
## Reproduction and focused checks
|
||||
|
||||
From this directory, apply the patch to a disposable SciPy 1.18.0 checkout:
|
||||
|
||||
```sh
|
||||
patch --fuzz=0 -d /path/to/scipy -p1 < stft-test-tolerances.patch
|
||||
```
|
||||
|
||||
Build and install that tree with SciPy's test dependencies. From outside the
|
||||
source directory, run the installed tests:
|
||||
|
||||
```sh
|
||||
python -m pytest --pyargs scipy.signal.tests.test_spectral \
|
||||
-k 'roundtrip_float32 or roundtrip_scaling' -q
|
||||
```
|
||||
|
||||
Use the same compiler flags and numerical libraries for before/after runs.
|
||||
The earlier reproduction called `TestSTFT.test_roundtrip_float32` and
|
||||
`TestSTFT.test_roundtrip_scaling` against the x86-64-v3 libraries, then loaded
|
||||
patched copies of the test modules. Both failed with the original bounds
|
||||
and passed with the adjusted bounds.
|
||||
|
||||
## Upstream status
|
||||
|
||||
[SciPy issue #25488](https://github.com/scipy/scipy/issues/25488) records
|
||||
related test failures with architecture-specific compiler flags. It is
|
||||
context for the local tolerance repair; this exact patch has not been
|
||||
submitted upstream during this work.
|
||||
|
||||
## Local NixOS integration and build results
|
||||
|
||||
[`../default.nix`](../default.nix) loads [`default.nix`](default.nix) through
|
||||
`pythonPackagesExtensions`, preserving the package's existing patches.
|
||||
The override also covers SciPy used to test other Python dependencies,
|
||||
including pgvector in portal's shared Python environment.
|
||||
|
||||
From the repository root:
|
||||
|
||||
```sh
|
||||
nix build --no-link -L .#nixosConfigurations.portal-1.pkgs.python314Packages.scipy
|
||||
```
|
||||
|
||||
The original remote build of patched SciPy 1.18.0 passed 87,723 tests, with
|
||||
8,342 skips, 300 expected failures, and 22 unexpected passes. The patch and
|
||||
override have been restored byte-for-byte from commit `24cbf74f`; those counts
|
||||
describe the earlier full build.
|
||||
|
||||
Restoration checks confirmed that the patch applies to the pinned source
|
||||
without fuzz, portal's evaluated SciPy retains its existing patch and install
|
||||
checks, and pgvector uses the patched SciPy. A full package or system rebuild
|
||||
was not repeated for this restoration.
|
||||
@@ -0,0 +1,5 @@
|
||||
{ scipy }:
|
||||
scipy.overridePythonAttrs (old: {
|
||||
# Keep the STFT tests enabled with tolerances for x86-64-v3 rounding.
|
||||
patches = (old.patches or [ ]) ++ [ ./stft-test-tolerances.patch ];
|
||||
})
|
||||
@@ -0,0 +1,51 @@
|
||||
Subject: [PATCH] signal: allow floating-point rounding in STFT tests
|
||||
|
||||
Keep the STFT tests enabled for x86-64-v3 builds. Allow two float32
|
||||
epsilons of relative error when comparing inverse-STFT implementations;
|
||||
float64 and the existing i686 override remain unchanged. Allow one
|
||||
float64 epsilon of absolute error in all three scaling round trips,
|
||||
which otherwise require exact zeros (observed residual: 4e-17 for a
|
||||
signal with amplitude 2).
|
||||
|
||||
Upstream issue: https://github.com/scipy/scipy/issues/25488
|
||||
|
||||
--- a/scipy/signal/tests/_scipy_spectral_test_shim.py
|
||||
+++ b/scipy/signal/tests/_scipy_spectral_test_shim.py
|
||||
@@ -294,7 +294,7 @@
|
||||
|
||||
# Adapted tolerances to account for resolution loss:
|
||||
atol = np.finfo(x.dtype).resolution*2 # instead of default atol = 0
|
||||
- rtol = 1e-7 # default for np.allclose()
|
||||
+ rtol = max(1e-7, 2 * np.finfo(x.dtype).eps)
|
||||
|
||||
# Relax atol on 32-Bit platforms a bit to pass CI tests.
|
||||
# - Not clear why there are discrepancies (in the FFT maybe?)
|
||||
--- a/scipy/signal/tests/test_spectral.py
|
||||
+++ b/scipy/signal/tests/test_spectral.py
|
||||
@@ -2044,7 +2044,7 @@
|
||||
|
||||
# Test round trip:
|
||||
x1 = istft(Zs, boundary=True, scaling='spectrum')[1]
|
||||
- assert_allclose(x1, x)
|
||||
+ assert_allclose(x1, x, atol=np.finfo(x.dtype).eps)
|
||||
|
||||
# For a Hann-windowed 256 sample length FFT, we expect a peak at
|
||||
# frequency 64 (since it is 1/4 the length of X) with a height of 1
|
||||
@@ -2074,7 +2074,7 @@
|
||||
|
||||
# Test round trip:
|
||||
x1 = istft(Zp, input_onesided=False, boundary=True, scaling='psd')[1]
|
||||
- assert_allclose(x1, x)
|
||||
+ assert_allclose(x1, x, atol=np.finfo(x.dtype).eps)
|
||||
|
||||
# The power of the one-sided psd-scaled STFT can be determined
|
||||
# analogously (note that the two sides are not of equal shape):
|
||||
@@ -2094,7 +2094,7 @@
|
||||
|
||||
# Test round trip:
|
||||
x1 = istft(Zp0, input_onesided=True, boundary=True, scaling='psd')[1]
|
||||
- assert_allclose(x1, x)
|
||||
+ assert_allclose(x1, x, atol=np.finfo(x.dtype).eps)
|
||||
|
||||
|
||||
class TestSampledSpectralRepresentations:
|
||||
@@ -0,0 +1,58 @@
|
||||
# Sentry SDK thread-metadata test isolation
|
||||
|
||||
The fallback tests globally mock `threading.current_thread` while a worker
|
||||
is running. Python 3.14's `Thread.join()` also calls that function. A one-use
|
||||
mock can therefore be consumed by the wrong caller or raise `StopIteration`
|
||||
when the main thread joins the worker.
|
||||
|
||||
## Scope and behavior
|
||||
|
||||
`isolate-threading-mocks.patch` changes three neighboring thread-metadata
|
||||
tests in `tests/test_utils.py`, including the formerly excluded
|
||||
`test_get_current_thread_meta_main_thread`.
|
||||
|
||||
Each test replaces only `sentry_sdk.utils.threading`, wraps the real module
|
||||
for unmocked operations, and sets the SDK lookup's return value. The real
|
||||
`Thread.join()` continues using Python's unmodified `threading` module.
|
||||
The fallback-result assertions remain; SDK production code is unchanged.
|
||||
|
||||
## Reproduction and focused checks
|
||||
|
||||
Use a disposable Sentry SDK 2.66.0 checkout and its Python test dependencies.
|
||||
From this directory:
|
||||
|
||||
```sh
|
||||
patch --fuzz=0 -d /path/to/sentry-python -p1 < isolate-threading-mocks.patch
|
||||
cd /path/to/sentry-python
|
||||
python -m pytest tests/test_utils.py -k get_current_thread_meta -q
|
||||
```
|
||||
|
||||
To reproduce the race, hold the worker inside its mock just after
|
||||
`get_current_thread_meta()` returns, signal that point to the main thread,
|
||||
and call `Thread.join()` before releasing the worker. Use an independent
|
||||
bounded release so the patched join can finish. The original test raises
|
||||
`StopIteration` in `join`; the patched test passes under the same schedule.
|
||||
Perform this scheduling instrumentation only in a disposable checkout.
|
||||
|
||||
## Upstream status
|
||||
|
||||
This is a standalone test patch for Sentry SDK 2.66.0. No upstream submission
|
||||
was made during this work. Recheck mock isolation and Python threading
|
||||
behavior when upgrading the SDK or interpreter.
|
||||
|
||||
## Local NixOS integration and build results
|
||||
|
||||
[`../default.nix`](../default.nix) loads `default.nix` through
|
||||
`pythonPackagesExtensions`. From the repository root:
|
||||
|
||||
```sh
|
||||
nix build --no-link -L .#nixosConfigurations.jeeves.pkgs.python314Packages.sentry-sdk
|
||||
```
|
||||
|
||||
The patched package passed 2,356 tests with 116 existing skips on Python
|
||||
3.14.7. The controlled join reproduction failed before the fix and passed
|
||||
after it.
|
||||
|
||||
That package build used the preceding dependency set with this patch to avoid
|
||||
unrelated rebuilds after pytest-xdist changed. The integrated host derivation
|
||||
was evaluated; a complete NixOS rebuild was not performed.
|
||||
@@ -0,0 +1,4 @@
|
||||
{ sentry-sdk }:
|
||||
sentry-sdk.overridePythonAttrs (old: {
|
||||
patches = (old.patches or [ ]) ++ [ ./isolate-threading-mocks.patch ];
|
||||
})
|
||||
@@ -0,0 +1,41 @@
|
||||
Subject: [PATCH] tests: isolate SDK thread lookup mocks from Python threading
|
||||
|
||||
Thread.join also calls threading.current_thread on Python 3.14. A global
|
||||
single-use side effect can be consumed by join instead of the SDK, or
|
||||
raise StopIteration in join after the SDK consumes it. Patch the SDK's
|
||||
module binding and delegate unmocked operations to the real module.
|
||||
Apply the same isolation to the adjacent invalid-thread fallback tests.
|
||||
|
||||
--- a/tests/test_utils.py
|
||||
+++ b/tests/test_utils.py
|
||||
@@ -914,7 +914,8 @@
|
||||
results = Queue(maxsize=1)
|
||||
|
||||
def target():
|
||||
- with mock.patch("threading.current_thread", side_effect=["fake thread"]):
|
||||
+ with mock.patch("sentry_sdk.utils.threading", wraps=threading) as sdk_threading:
|
||||
+ sdk_threading.current_thread.return_value = "fake thread"
|
||||
results.put(get_current_thread_meta())
|
||||
|
||||
thread = threading.Thread(target=target)
|
||||
@@ -930,7 +931,9 @@
|
||||
|
||||
def target():
|
||||
# mock that somehow the current thread doesn't exist
|
||||
- with mock.patch("threading.current_thread", side_effect=[None]):
|
||||
+ # Keep the real threading module intact for concurrent Thread.join calls.
|
||||
+ with mock.patch("sentry_sdk.utils.threading", wraps=threading) as sdk_threading:
|
||||
+ sdk_threading.current_thread.return_value = None
|
||||
results.put(get_current_thread_meta())
|
||||
|
||||
main_thread = threading.main_thread()
|
||||
@@ -945,7 +948,8 @@
|
||||
results = Queue(maxsize=1)
|
||||
|
||||
def target():
|
||||
- with mock.patch("threading.current_thread", return_value="fake thread"):
|
||||
+ with mock.patch("sentry_sdk.utils.threading", wraps=threading) as sdk_threading:
|
||||
+ sdk_threading.current_thread.return_value = "fake thread"
|
||||
results.put(get_current_thread_meta())
|
||||
|
||||
main_thread = threading.main_thread()
|
||||
@@ -1,133 +0,0 @@
|
||||
# Test exclusions for the locally rebuilt x86-64-v3 package set.
|
||||
#
|
||||
# Selecting x86-64-v3 changes every affected derivation, so the normal
|
||||
# nixpkgs binary cache cannot be used and upstream test suites run locally.
|
||||
# The jeeves builder uses /tmp/nix-builds so filesystem tests run on tmpfs
|
||||
# instead of ZFS with normalization=formD and utf8only=on; those tests remain
|
||||
# enabled. The remaining workarounds cover UDP readiness, resource-sensitive
|
||||
# parser and nested-worker races, and mismatched timeout clocks, plus
|
||||
# architecture-dependent floating-point differences whose risk we accept for
|
||||
# our workloads. Keep these exceptions visible until their causes are fixed.
|
||||
_final: prev: {
|
||||
gnutls = prev.gnutls.overrideAttrs (old: {
|
||||
# This test uses a fixed four-second sleep instead of checking UDP
|
||||
# readiness; the client saw no listener in the x86-64-v3 build.
|
||||
postPatch = (old.postPatch or "") + ''
|
||||
sed '2iexit 77' -i tests/serv-udp.sh
|
||||
'';
|
||||
});
|
||||
|
||||
prometheus = prev.prometheus.overrideAttrs (
|
||||
old:
|
||||
let
|
||||
assets = old.passthru.assets.overrideAttrs (assetsOld: {
|
||||
# CodeMirror's bounded synchronous parser can return an incomplete tree
|
||||
# when these cases run on a heavily loaded builder.
|
||||
postPatch = (assetsOld.postPatch or "") + ''
|
||||
substituteInPlace module/codemirror-promql/src/complete/hybrid.test.ts \
|
||||
--replace-fail "it(value.title, () => {" \
|
||||
"(value.title === 'autocomplete topk params 2' ? it.skip : it)(value.title, () => {"
|
||||
substituteInPlace module/codemirror-promql/src/parser/vector.test.ts \
|
||||
--replace-fail "it(value.binaryExpr, () => {" \
|
||||
"(value.binaryExpr === 'foo * on(test,blub) bar' ? it.skip : it)(value.binaryExpr, () => {"
|
||||
'';
|
||||
});
|
||||
in
|
||||
{
|
||||
postPatch = builtins.replaceStrings [ "${old.passthru.assets}" ] [ "${assets}" ] (
|
||||
builtins.unsafeDiscardStringContext old.postPatch
|
||||
);
|
||||
passthru = old.passthru // {
|
||||
inherit assets;
|
||||
};
|
||||
}
|
||||
);
|
||||
|
||||
pythonPackagesExtensions = prev.pythonPackagesExtensions ++ [
|
||||
(_pythonFinal: pythonPrev: {
|
||||
backrefs = pythonPrev.backrefs.overridePythonAttrs (old: {
|
||||
# regex measures its timeout in process CPU time, while this test used
|
||||
# wall time and could miss the timeout when a busy builder descheduled it.
|
||||
postPatch = (old.postPatch or "") + ''
|
||||
substituteInPlace tests/test_bregex.py \
|
||||
--replace-fail "time.time()" "time.process_time()"
|
||||
'';
|
||||
});
|
||||
|
||||
pytest-xdist = pythonPrev.pytest-xdist.overridePythonAttrs (old: {
|
||||
# The suite exercises its own worker pools. Run the outer suite with one
|
||||
# worker and allow inner workers more time on heavily loaded builders.
|
||||
postPatch = (old.postPatch or "") + ''
|
||||
substituteInPlace testing/test_remote.py \
|
||||
--replace-fail "WAIT_TIMEOUT = 10.0" "WAIT_TIMEOUT = 60.0"
|
||||
'';
|
||||
preCheck = builtins.replaceStrings [ "--numprocesses=$NIX_BUILD_CORES" ] [ "--numprocesses=1" ] (
|
||||
old.preCheck or ""
|
||||
);
|
||||
# This test deliberately crashes workers past the restart limit and
|
||||
# races while checking which replacement message was emitted.
|
||||
disabledTests = (old.disabledTests or [ ]) ++ [
|
||||
"test_max_worker_restart_tests_queued"
|
||||
];
|
||||
});
|
||||
|
||||
jupyter-server = pythonPrev.jupyter-server.overridePythonAttrs (old: {
|
||||
# The kernel reply arrived after the one-second outer deadline when the
|
||||
# builder was heavily loaded. Keep the regression test but allow it the
|
||||
# same margin as the other resource-sensitive tests.
|
||||
postPatch = (old.postPatch or "") + ''
|
||||
substituteInPlace tests/services/kernels/test_connection.py \
|
||||
--replace-fail \
|
||||
"await asyncio.wait_for(asyncio.wrap_future(conn2.request_kernel_info()), timeout=1.0)" \
|
||||
"await asyncio.wait_for(asyncio.wrap_future(conn2.request_kernel_info()), timeout=10.0)"
|
||||
'';
|
||||
});
|
||||
|
||||
scipy = pythonPrev.scipy.overridePythonAttrs (old: {
|
||||
# x86-64-v3 FFT implementations produce rounding differences outside
|
||||
# these tests' strict tolerances. We accept the numerical-precision
|
||||
# risk for our workloads.
|
||||
disabledTests = (old.disabledTests or [ ]) ++ [
|
||||
"test_roundtrip_float32"
|
||||
"test_roundtrip_scaling"
|
||||
];
|
||||
});
|
||||
|
||||
sentry-sdk = pythonPrev.sentry-sdk.overridePythonAttrs (old: {
|
||||
# This test globally mocks threading.current_thread while another
|
||||
# thread is running. On Python 3.14, Thread.join can race with that
|
||||
# mock and exhaust its single side effect before the worker removes it.
|
||||
disabledTests = (old.disabledTests or [ ]) ++ [
|
||||
"test_get_current_thread_meta_main_thread"
|
||||
];
|
||||
});
|
||||
|
||||
torchaudio = pythonPrev.torchaudio.overridePythonAttrs (old: {
|
||||
# x86-64-v3 pitch shifting produces batch-versus-single-item numerical
|
||||
# differences up to 2.9e-6. We accept that audio-precision risk for our
|
||||
# workloads.
|
||||
disabledTests = (old.disabledTests or [ ]) ++ [
|
||||
"test_batch_pitch_shift"
|
||||
];
|
||||
});
|
||||
|
||||
torchcodec = pythonPrev.torchcodec.overridePythonAttrs (old: {
|
||||
# For these 8 kHz MP3 cases, the x86-64-v3 API and CLI codec paths
|
||||
# differ in 0.8% of decoded samples. Retain the original tolerance for
|
||||
# 99% of samples and accept the localized audio-precision risk.
|
||||
postPatch = (old.postPatch or "") + ''
|
||||
substituteInPlace test/test_encoders.py \
|
||||
--replace-fail \
|
||||
'if sys.platform == "darwin":' \
|
||||
'if sys.platform == "darwin" or (
|
||||
format == "mp3"
|
||||
and sample_rate == 8_000
|
||||
and asset is SINE_MONO_S32
|
||||
and bit_rate in (None, 0)
|
||||
and num_channels in (None, 1)
|
||||
):'
|
||||
'';
|
||||
});
|
||||
})
|
||||
];
|
||||
}
|
||||
@@ -1,60 +0,0 @@
|
||||
# Compatibility fixes for packages rebuilt with x86-64-v3.
|
||||
#
|
||||
# The v3 baseline enables instructions that expose source assumptions hidden
|
||||
# by the generic x86-64 build. Keep compile fixes here, separate from test
|
||||
# exclusions, until upstream or nixpkgs incorporates them.
|
||||
_final: prev:
|
||||
let
|
||||
patchAbseilBmi2Include =
|
||||
package:
|
||||
package.overrideAttrs (old: {
|
||||
# GCC and Clang prohibit including their internal BMI2 header directly.
|
||||
# The public umbrella provides the same intrinsics with the required
|
||||
# compiler setup.
|
||||
postPatch = (old.postPatch or "") + ''
|
||||
substituteInPlace third_party/abseil-cpp/absl/container/internal/raw_hash_set.h \
|
||||
--replace-fail "#include <bmi2intrin.h>" "#include <immintrin.h>"
|
||||
'';
|
||||
});
|
||||
|
||||
removeSiblingOutputChecks =
|
||||
package:
|
||||
package.overrideAttrs (old: {
|
||||
# Nix 2.34 can validate a partial multi-output rebuild against only the
|
||||
# outputs still being realised. PostgreSQL's checks then reject valid
|
||||
# sibling names such as "out" and "lib". Keep the test suite and
|
||||
# disallowed-requisite checks; accept the loss of cross-output checks.
|
||||
outputChecks = builtins.mapAttrs (
|
||||
_output: checks: builtins.removeAttrs checks [ "disallowedReferences" ]
|
||||
) (old.outputChecks or { });
|
||||
});
|
||||
|
||||
electron43Unwrapped = patchAbseilBmi2Include prev.electron_43.unwrapped;
|
||||
electron43 = prev.electron_43.override {
|
||||
electron-unwrapped = electron43Unwrapped;
|
||||
};
|
||||
|
||||
signalCallPackage =
|
||||
path: args:
|
||||
let
|
||||
package = prev.callPackage path args;
|
||||
in
|
||||
if builtins.baseNameOf path == "webrtc.nix" then patchAbseilBmi2Include package else package;
|
||||
in
|
||||
prev.lib.optionalAttrs ((prev.stdenv.hostPlatform.gcc.arch or null) == "x86-64-v3") {
|
||||
deno =
|
||||
let
|
||||
librusty_v8 = patchAbseilBmi2Include prev.deno.passthru.librusty_v8;
|
||||
in
|
||||
prev.deno.override { inherit librusty_v8; };
|
||||
|
||||
electron_43 = electron43;
|
||||
|
||||
postgresql = removeSiblingOutputChecks prev.postgresql;
|
||||
postgresql_18 = removeSiblingOutputChecks prev.postgresql_18;
|
||||
|
||||
signal-desktop = prev.signal-desktop.override {
|
||||
electron_43 = electron43;
|
||||
callPackage = signalCallPackage;
|
||||
};
|
||||
}
|
||||
@@ -21,5 +21,16 @@
|
||||
nix-builder-12.enable = true;
|
||||
nix-builder-13.enable = true;
|
||||
nix-builder-14.enable = true;
|
||||
|
||||
# Warm the shared x86-64-v3 cache before the smaller per-system runners
|
||||
# start. Eight jobs with eight cores each can use Jeeves' 64 logical CPUs,
|
||||
# while the 6000% quota leaves some capacity for its normal services.
|
||||
nix-cache-builder = {
|
||||
enable = true;
|
||||
labels = [ "nix-cache-builder:host" ];
|
||||
cores = 8;
|
||||
maxJobs = 8;
|
||||
cpuQuota = "6000%";
|
||||
};
|
||||
};
|
||||
}
|
||||
|
||||
@@ -11,11 +11,8 @@ let
|
||||
cfg = config.services.nix_builder;
|
||||
runnerUsername = "gitea-runner";
|
||||
runnerUserid = 601;
|
||||
runnerLabels = [
|
||||
"self-hosted:host"
|
||||
"nixos:host"
|
||||
];
|
||||
containerConfig =
|
||||
containerCfg:
|
||||
{
|
||||
config,
|
||||
pkgs,
|
||||
@@ -50,8 +47,8 @@ let
|
||||
useHostResolvConf = false;
|
||||
};
|
||||
nix.settings = {
|
||||
cores = 8;
|
||||
max-jobs = 2;
|
||||
inherit (containerCfg) cores;
|
||||
max-jobs = containerCfg.maxJobs;
|
||||
system-features = lib.mkAfter [
|
||||
"gccarch-x86-64-v2"
|
||||
"gccarch-x86-64-v3"
|
||||
@@ -94,7 +91,7 @@ let
|
||||
enable = true;
|
||||
name = "jeeves-nix-builder";
|
||||
url = "http://192.168.99.14:6443/";
|
||||
labels = runnerLabels;
|
||||
labels = containerCfg.labels;
|
||||
tokenFile = "/run/secrets/gitea-runners/registration-token";
|
||||
settings.runner.timeout = "12h";
|
||||
hostPackages = with pkgs; [
|
||||
@@ -120,20 +117,21 @@ let
|
||||
User = mkForce runnerUsername;
|
||||
Group = mkForce runnerUsername;
|
||||
ExecStartPre = mkForce [
|
||||
"${getExe registerRunner} builder http://192.168.99.14:6443/ ${runnerConfigFile} ${escapeShellArgs runnerLabels}"
|
||||
"${getExe registerRunner} builder http://192.168.99.14:6443/ ${runnerConfigFile} ${escapeShellArgs containerCfg.labels}"
|
||||
];
|
||||
};
|
||||
};
|
||||
system.stateVersion = "24.05";
|
||||
};
|
||||
sharedContainerPath =
|
||||
mkContainerPath =
|
||||
containerCfg:
|
||||
(import "${pkgs.path}/nixos/lib/eval-config.nix" {
|
||||
modules = [
|
||||
{
|
||||
boot.isNspawnContainer = true;
|
||||
nixpkgs.pkgs = pkgs;
|
||||
}
|
||||
containerConfig
|
||||
(containerConfig containerCfg)
|
||||
];
|
||||
system = null;
|
||||
}).config.system.build.toplevel;
|
||||
@@ -151,7 +149,36 @@ in
|
||||
types.submodule (
|
||||
{ name, ... }:
|
||||
{
|
||||
options.enable = mkEnableOption "Gitea runner container";
|
||||
options = {
|
||||
enable = mkEnableOption "Gitea runner container";
|
||||
|
||||
labels = mkOption {
|
||||
type = types.listOf types.str;
|
||||
default = [
|
||||
"self-hosted:host"
|
||||
"nixos:host"
|
||||
];
|
||||
description = "Gitea Actions labels advertised by this runner.";
|
||||
};
|
||||
|
||||
cores = mkOption {
|
||||
type = types.ints.positive;
|
||||
default = 8;
|
||||
description = "Number of cores made available to each Nix build job.";
|
||||
};
|
||||
|
||||
maxJobs = mkOption {
|
||||
type = types.ints.positive;
|
||||
default = 2;
|
||||
description = "Maximum number of Nix build jobs run in parallel.";
|
||||
};
|
||||
|
||||
cpuQuota = mkOption {
|
||||
type = types.str;
|
||||
default = "800%";
|
||||
description = "systemd CPU quota for the runner container.";
|
||||
};
|
||||
};
|
||||
}
|
||||
)
|
||||
);
|
||||
@@ -173,7 +200,7 @@ in
|
||||
containers = mapAttrs (
|
||||
name: containerCfg:
|
||||
mkIf containerCfg.enable {
|
||||
path = sharedContainerPath;
|
||||
path = mkContainerPath containerCfg;
|
||||
autoStart = true;
|
||||
privateNetwork = true;
|
||||
hostBridge = cfg.bridgeName;
|
||||
@@ -199,16 +226,14 @@ in
|
||||
) cfg.containers;
|
||||
|
||||
systemd = {
|
||||
services = builtins.listToAttrs (
|
||||
map (name: {
|
||||
name = "container@${name}";
|
||||
value = {
|
||||
requires = [ "gitea.service" ];
|
||||
after = [ "gitea.service" ];
|
||||
serviceConfig.CPUQuota = "800%";
|
||||
};
|
||||
}) (builtins.attrNames (filterAttrs (_: c: c.enable) cfg.containers))
|
||||
);
|
||||
services = mapAttrs' (
|
||||
name: containerCfg:
|
||||
nameValuePair "container@${name}" {
|
||||
requires = [ "gitea.service" ];
|
||||
after = [ "gitea.service" ];
|
||||
serviceConfig.CPUQuota = containerCfg.cpuQuota;
|
||||
}
|
||||
) (filterAttrs (_: c: c.enable) cfg.containers);
|
||||
|
||||
tmpfiles.rules = [
|
||||
"d ${vars.uv_cache} 0755 ${runnerUsername} ${runnerUsername} - -"
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{ pkgs, ... }:
|
||||
{
|
||||
home.packages = with pkgs; [
|
||||
home.packages = with pkgs.x86-v1; [
|
||||
discord-canary
|
||||
signal-desktop
|
||||
slack
|
||||
|
||||
@@ -14,14 +14,13 @@
|
||||
gimp
|
||||
mediainfo
|
||||
obs-studio
|
||||
obsidian
|
||||
x86-v1.obsidian
|
||||
prismlauncher
|
||||
prusa-slicer
|
||||
qalculate-gtk
|
||||
vlc
|
||||
# browser
|
||||
brave
|
||||
chromium
|
||||
# dev tools
|
||||
gparted
|
||||
jetbrains.datagrip
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{ pkgs, ... }:
|
||||
{
|
||||
home.packages = with pkgs; [
|
||||
home.packages = with pkgs.x86-v1; [
|
||||
sweet-nova
|
||||
candy-icons
|
||||
];
|
||||
|
||||
@@ -13,7 +13,7 @@ in
|
||||
|
||||
programs.vscode = {
|
||||
enable = true;
|
||||
package = pkgs.vscode;
|
||||
package = pkgs.x86-v1.vscode;
|
||||
mutableExtensionsDir = true;
|
||||
};
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user