feat(haproxy): add initial implementation for HAProxy log ingestion

rust/haproxy_ingest_proto/README.md

@@ -0,0 +1,96 @@
+# HAProxy cloud log ingestion
+
+This workspace is split into three crates:
+
+- `haproxy-ingest-proto` owns the shared Protobuf schema and protocol limits.
+- `haproxy-log-shipper` is a short-lived client launched by a systemd timer on
+  `portal-1`. It reads only local `haproxy.service` journal records and drains
+  them in cursor order.
+- `haproxy-ingest-server` is the private HTTP API on Jeeves. It authenticates
+  the source, commits records to PostgreSQL, and advances the journal cursor
+  atomically.
+
+The wire contract is in `proto/haproxy_ingest.proto`; the initial PostgreSQL
+schema is in `../haproxy_ingest_server/migrations/001_initial.sql`.
+
+## One-time setup
+
+The shipper endpoint is supplied by the `haproxy-log-shipper` service as
+`http://jeeves.taile39c31.ts.net:8444`; the environment file only needs the
+shared token. Ingestion uses HTTP over Tailscale and requires no certificates.
+
+Create one random token and install the same value on the shipper and server.
+A VM rebuild with a new `/etc/machine-id` is still treated as a new log source.
+
+On `portal-1`, create `/var/lib/haproxy-log-shipper/environment`:
+
+```text
+HAPROXY_INGEST_TOKEN=<at-least-32-character-random-token>
+```
+
+Set the file ownership and mode to `root:root` and `0600`.
+
+On Jeeves, create
+`/zfs/storage/secrets/services/haproxy-ingest-server`:
+
+```text
+HAPROXY_INGEST_TOKEN=<the-same-random-token>
+```
+
+Set this file to `root:root` and `0600`. systemd loads it before switching the
+API to its unprivileged service account.
+
+A suitable token can be generated without writing it to shell history:
+
+```console
+openssl rand -hex 32
+```
+
+Deploy Jeeves before `portal-1`, then verify:
+
+```console
+systemctl status haproxy-ingest-server.service
+systemctl status haproxy-log-shipper.timer
+journalctl -u haproxy-log-shipper.service
+```
+
+The server listens over IPv4 on TCP 8444, which Jeeves' firewall permits only
+through `tailscale0`. Both service sandboxes disable IPv6. Tailscale encrypts
+traffic between the hosts. The bearer token authorizes access to the ingestion API;
+the machine ID identifies a log source but is not an authorization boundary.
+
+## Protocol behavior
+
+- Maximum sender batch: 1,000 records or 20 MiB serialized Protobuf.
+- Maximum receiver request: 24 MiB compressed, 20 MiB decompressed, 10,000 records.
+- Compression: one Zstandard frame over the complete Protobuf body.
+- Request timeouts: 3 seconds to connect and 10 seconds total.
+- Retry delays: 1, 2, and 4 seconds plus bounded jitter.
+- `409`: fetch the authoritative server cursor and rebuild.
+- `413`: fail because a sender batch within the configured limits was rejected.
+- `422`: replace only the reported offending record with a durable rejection.
+- Ambiguous failures: query the status endpoint before retrying.
+
+Journal retention on `portal-1` is persistent, bounded to 512 MiB, and retained
+for up to 14 days. If an acknowledged cursor has already been vacuumed, the
+shipper records a `JournalGap` before resuming from the oldest retained matching
+entry.
+
+## Database inspection
+
+The API owns the `haproxy_ingest` database and creates daily
+`haproxy_logs_YYYYMMDD` partitions before using chunked SQLx inserts.
+Useful checks include:
+
+```sql
+SELECT machine_id, hostname, last_journal_timestamp, last_commit_time
+FROM ingest_source_state;
+
+SELECT journal_timestamp, client_ip, http_method, request_path, status_code
+FROM haproxy_logs
+ORDER BY journal_timestamp DESC
+LIMIT 100;
+
+SELECT * FROM ingest_journal_gaps ORDER BY detected_at DESC;
+SELECT * FROM ingest_rejected_events ORDER BY ingest_time DESC;
+```