Worker Kill Timestamp
2026-03-08T18:49:31.938Z
Reliability Evidence
Run proofpack-20260308T184847Z
NetPulse Proof Pack
Execution window (UTC): 2026-03-08T18:48:47Z to 2026-03-08T18:49:36Z
Hypothesis: if 50% of regional workers fail during a 10k-event burst, NetPulse should detect quickly, replay queued events, and keep total data loss at zero.
Commands
docker compose -f infra/high-concurrency/docker-compose.yml up -d
TOTAL_WRITES=10000 WRITE_CONCURRENCY=1000 POOL_MAX_CONNECTIONS=1200 npm run drill:pgbouncer
MTLS_WORKERS=4 MTLS_EVENTS_PER_WORKER=2500 MTLS_EVENT_INTERVAL_MS=2 MTLS_KILL_RATIO=0.5 MTLS_KILL_AFTER_MS=1500 npm run drill:mtls:chaos
chaos-injector kill regional-worker-1 regional-worker-2
Detection Latency
5ms
Data Loss
0
Processed / Expected
10000/10000
DLQ Replay
3684/3684
PgBouncer Spike
10000 writes
failed 0 | 5564.83/sec
Postgres Connections
1/17 active/total
Grafana Evidence Points
- Worker health drops from 4 to 2 at kill timestamp.
- Security/error signal spikes on mTLS handshake rejection.
- Processed jobs curve continues climbing after the outage.
- DLQ queued and replayed converge to equal counts.