97 lines
3.5 KiB
Markdown
97 lines
3.5 KiB
Markdown
# HAProxy cloud log ingestion
|
|
|
|
This workspace is split into three crates:
|
|
|
|
- `haproxy-ingest-proto` owns the shared Protobuf schema and protocol limits.
|
|
- `haproxy-log-shipper` is a short-lived client launched by a systemd timer on
|
|
`portal-1`. It reads only local `haproxy.service` journal records and drains
|
|
them in cursor order.
|
|
- `haproxy-ingest-server` is the private HTTP API on Jeeves. It authenticates
|
|
the source, commits records to PostgreSQL, and advances the journal cursor
|
|
atomically.
|
|
|
|
The wire contract is in `proto/haproxy_ingest.proto`; the initial PostgreSQL
|
|
schema is in `../haproxy_ingest_server/migrations/001_initial.sql`.
|
|
|
|
## One-time setup
|
|
|
|
The shipper endpoint is supplied by the `haproxy-log-shipper` service as
|
|
`http://jeeves.taile39c31.ts.net:8444`; the environment file only needs the
|
|
shared token. Ingestion uses HTTP over Tailscale and requires no certificates.
|
|
|
|
Create one random token and install the same value on the shipper and server.
|
|
A VM rebuild with a new `/etc/machine-id` is still treated as a new log source.
|
|
|
|
On `portal-1`, create `/var/lib/haproxy-log-shipper/environment`:
|
|
|
|
```text
|
|
HAPROXY_INGEST_TOKEN=<at-least-32-character-random-token>
|
|
```
|
|
|
|
Set the file ownership and mode to `root:root` and `0600`.
|
|
|
|
On Jeeves, create
|
|
`/zfs/storage/secrets/services/haproxy-ingest-server`:
|
|
|
|
```text
|
|
HAPROXY_INGEST_TOKEN=<the-same-random-token>
|
|
```
|
|
|
|
Set this file to `root:root` and `0600`. systemd loads it before switching the
|
|
API to its unprivileged service account.
|
|
|
|
A suitable token can be generated without writing it to shell history:
|
|
|
|
```console
|
|
openssl rand -hex 32
|
|
```
|
|
|
|
Deploy Jeeves before `portal-1`, then verify:
|
|
|
|
```console
|
|
systemctl status haproxy-ingest-server.service
|
|
systemctl status haproxy-log-shipper.timer
|
|
journalctl -u haproxy-log-shipper.service
|
|
```
|
|
|
|
The server listens over IPv4 on TCP 8444, which Jeeves' firewall permits only
|
|
through `tailscale0`. Both service sandboxes disable IPv6. Tailscale encrypts
|
|
traffic between the hosts. The bearer token authorizes access to the ingestion API;
|
|
the machine ID identifies a log source but is not an authorization boundary.
|
|
|
|
## Protocol behavior
|
|
|
|
- Maximum sender batch: 1,000 records or 20 MiB serialized Protobuf.
|
|
- Maximum receiver request: 24 MiB compressed, 20 MiB decompressed, 10,000 records.
|
|
- Compression: one Zstandard frame over the complete Protobuf body.
|
|
- Request timeouts: 3 seconds to connect and 10 seconds total.
|
|
- Retry delays: 1, 2, and 4 seconds plus bounded jitter.
|
|
- `409`: fetch the authoritative server cursor and rebuild.
|
|
- `413`: fail because a sender batch within the configured limits was rejected.
|
|
- `422`: replace only the reported offending record with a durable rejection.
|
|
- Ambiguous failures: query the status endpoint before retrying.
|
|
|
|
Journal retention on `portal-1` is persistent, bounded to 512 MiB, and retained
|
|
for up to 14 days. If an acknowledged cursor has already been vacuumed, the
|
|
shipper records a `JournalGap` before resuming from the oldest retained matching
|
|
entry.
|
|
|
|
## Database inspection
|
|
|
|
The API owns the `haproxy_ingest` database and creates daily
|
|
`haproxy_logs_YYYYMMDD` partitions before using chunked SQLx inserts.
|
|
Useful checks include:
|
|
|
|
```sql
|
|
SELECT machine_id, hostname, last_journal_timestamp, last_commit_time
|
|
FROM ingest_source_state;
|
|
|
|
SELECT journal_timestamp, client_ip, http_method, request_path, status_code
|
|
FROM haproxy_logs
|
|
ORDER BY journal_timestamp DESC
|
|
LIMIT 100;
|
|
|
|
SELECT * FROM ingest_journal_gaps ORDER BY detected_at DESC;
|
|
SELECT * FROM ingest_rejected_events ORDER BY ingest_time DESC;
|
|
```
|