dotfiles/rust/haproxy_ingest_proto/README.md

HAProxy cloud log ingestion

This workspace is split into three crates:

• haproxy-ingest-proto owns the shared Protobuf schema and protocol limits.
• haproxy-log-shipper is a short-lived client launched by a systemd timer on portal-1. It reads only local haproxy.service journal records and drains them in cursor order.
• haproxy-ingest-server is the private HTTP API on Jeeves. It authenticates the source, commits records to PostgreSQL, and advances the journal cursor atomically.

The wire contract is in proto/haproxy_ingest.proto; the initial PostgreSQL schema is in ../haproxy_ingest_server/migrations/001_initial.sql.

One-time setup

The shipper endpoint is supplied by the haproxy-log-shipper service as http://jeeves.taile39c31.ts.net:8444; the environment file only needs the shared token. Ingestion uses HTTP over Tailscale and requires no certificates.

Create one random token and install the same value on the shipper and server. A VM rebuild with a new /etc/machine-id is still treated as a new log source.

On portal-1, create /var/lib/haproxy-log-shipper/environment:

HAPROXY_INGEST_TOKEN=<at-least-32-character-random-token>

Set the file ownership and mode to root:root and 0600.

On Jeeves, create /zfs/storage/secrets/services/haproxy-ingest-server:

HAPROXY_INGEST_TOKEN=<the-same-random-token>

Set this file to root:root and 0600. systemd loads it before switching the API to its unprivileged service account.

A suitable token can be generated without writing it to shell history:

openssl rand -hex 32

Deploy Jeeves before portal-1, then verify:

systemctl status haproxy-ingest-server.service
systemctl status haproxy-log-shipper.timer
journalctl -u haproxy-log-shipper.service

The server listens over IPv4 on TCP 8444, which Jeeves' firewall permits only through tailscale0. Both service sandboxes disable IPv6. Tailscale encrypts traffic between the hosts. The bearer token authorizes access to the ingestion API; the machine ID identifies a log source but is not an authorization boundary.

Protocol behavior

• Maximum sender batch: 1,000 records or 20 MiB serialized Protobuf.
• Maximum receiver request: 24 MiB compressed, 20 MiB decompressed, 10,000 records.
• Compression: one Zstandard frame over the complete Protobuf body.
• Request timeouts: 3 seconds to connect and 10 seconds total.
• Retry delays: 1, 2, and 4 seconds plus bounded jitter.
• 409: fetch the authoritative server cursor and rebuild.
• 413: fail because a sender batch within the configured limits was rejected.
• 422: replace only the reported offending record with a durable rejection.
• Ambiguous failures: query the status endpoint before retrying.

Journal retention on portal-1 is persistent, bounded to 512 MiB, and retained for up to 14 days. If an acknowledged cursor has already been vacuumed, the shipper records a JournalGap before resuming from the oldest retained matching entry.

Database inspection

The API owns the haproxy_ingest database and creates daily haproxy_logs_YYYYMMDD partitions before using chunked SQLx inserts. Useful checks include:

SELECT machine_id, hostname, last_journal_timestamp, last_commit_time
FROM ingest_source_state;

SELECT journal_timestamp, client_ip, http_method, request_path, status_code
FROM haproxy_logs
ORDER BY journal_timestamp DESC
LIMIT 100;

SELECT * FROM ingest_journal_gaps ORDER BY detected_at DESC;
SELECT * FROM ingest_rejected_events ORDER BY ingest_time DESC;