Inject failure into live services and prove the resilience patterns hold
AutoSRE is a chaos engineering and resilience platform that runs as a local Docker Compose stack. It injects latency, errors, CPU burn and memory pressure into live services through a REST control plane, then measures whether the resilience patterns actually hold. Experiments are bounded at two layers β module-level caps and per-request validation β and the control plane, health probes and metrics endpoints are exempt from injection, so the abort path can never be made unreachable.
Its guiding principle: observe, measure and notify β never auto-remediate. Nothing restarts, scales or heals itself. Failures stay visible instead of being masked by automatic recovery, and a human responds using the runbooks.
Scope, stated plainly: this is a local demonstration stack, not a production system, and it does not serve real users. It runs a single instance of each service, so there is no load balancer, no replicas and no failover. Alert rules evaluate and fire in the Prometheus UI; Alertmanager notification routing is deliberately left unwired.
REST endpoints inject latency, HTTP 500s, CPU burn and memory pressure into each service. CPU and memory experiments are time-limited and revert themselves; latency and error injection stay on until they are disabled or reset.
Errors are spread deterministically across requests with a Bresenham accumulator, and the memory injector touches one byte per 4KB page to force a real commit instead of a lazy allocation.
Closed, open and half-open, thread-safe, with an injectable clock for deterministic tests, a pluggable store interface and Prometheus state metrics.
Per-client rate limiting middleware returning HTTP 429 with a Retry-After header. It bounds request rate; in-flight concurrency is measured as a saturation signal, not capped.
Rate, Errors and Duration instrumented through ASGI middleware with bounded label cardinality β matched route templates, never raw paths. A p95 500ms SLO with a 750ms alert threshold, defined as Prometheus rules.
The alerting rules are unit-tested themselves: every alert must carry a severity, a summary, a description and a runbook link, and the runbook it points at has to exist.
Request and correlation IDs generated when absent, honoured when supplied, echoed on the response and forwarded across services β carried by pure ASGI middleware so the context propagates reliably.
A nine-job CI pipeline runs Ruff, gofmt, go vet, go test, mypy, pytest, Bandit, Trivy, pip-audit and govulncheck, plus Docker build, Prometheus and Grafana provisioning validation. Third-party actions are pinned to immutable commit SHAs.
A reusable Compose anchor applies read-only root filesystems, all Linux capabilities dropped, no-new-privileges, tmpfs /tmp, CPU and memory limits and a graceful stop period. Services run non-root; the Go daemon ships distroless.
An independent vantage point written in Go that probes DNS, TCP and HTTP reachability and latency, so "the network is broken" can be told apart from "the app is broken".
Seven operator runbooks, six architecture decision records and three game-day postmortems β each postmortem carrying the exact command that reproduces the incident.
59 Python tests and 19 Go tests, at 93.9% line and 85.5% branch coverage on the shared platform library.
A FastAPI service exporting RED metrics β rate, errors and duration β to Prometheus
Scrape configuration, recording rules, SLOs and error budgets, and four provisioned dashboards
api, auth and worker on one shared platform library, with circuit breakers and rate limiting
Bounded failure injection through a REST control plane, with the abort path held exempt
Game-day postmortems with timelines, root cause and a command to reproduce each incident
Container hardening, security scanning, dependency auditing and config validation in CI
Every design decision is recorded as an architecture decision record, every alert links to a runbook, and every incident has a postmortem carrying the command that reproduces it.
View on GitHub β