Chaos Engineering & Resilience Platform

Inject failure into live services and prove the resilience patterns hold

βœ… Complete Β· Feb – Jul 2026 πŸ” Observe & notify β€” never auto-remediate

What is AutoSRE?

AutoSRE is a chaos engineering and resilience platform that runs as a local Docker Compose stack. It injects latency, errors, CPU burn and memory pressure into live services through a REST control plane, then measures whether the resilience patterns actually hold. Experiments are bounded at two layers β€” module-level caps and per-request validation β€” and the control plane, health probes and metrics endpoints are exempt from injection, so the abort path can never be made unreachable.

Its guiding principle: observe, measure and notify β€” never auto-remediate. Nothing restarts, scales or heals itself. Failures stay visible instead of being masked by automatic recovery, and a human responds using the runbooks.

Scope, stated plainly: this is a local demonstration stack, not a production system, and it does not serve real users. It runs a single instance of each service, so there is no load balancer, no replicas and no failover. Alert rules evaluate and fire in the Prometheus UI; Alertmanager notification routing is deliberately left unwired.

What's Built βœ…

πŸ”₯ Chaos Control Plane

REST endpoints inject latency, HTTP 500s, CPU burn and memory pressure into each service. CPU and memory experiments are time-limited and revert themselves; latency and error injection stay on until they are disabled or reset.

🎯 Reproducible, Not Random

Errors are spread deterministically across requests with a Bresenham accumulator, and the memory injector touches one byte per 4KB page to force a real commit instead of a lazy allocation.

⚑ Three-State Circuit Breaker

Closed, open and half-open, thread-safe, with an injectable clock for deterministic tests, a pluggable store interface and Prometheus state metrics.

🚦 Token-Bucket Rate Limiting

Per-client rate limiting middleware returning HTTP 429 with a Retry-After header. It bounds request rate; in-flight concurrency is measured as a saturation signal, not capped.

πŸ“Š RED Metrics & SLOs

Rate, Errors and Duration instrumented through ASGI middleware with bounded label cardinality β€” matched route templates, never raw paths. A p95 500ms SLO with a 750ms alert threshold, defined as Prometheus rules.

πŸ§ͺ Alert Rules Under Test

The alerting rules are unit-tested themselves: every alert must carry a severity, a summary, a description and a runbook link, and the runbook it points at has to exist.

πŸ”— Correlation IDs

Request and correlation IDs generated when absent, honoured when supplied, echoed on the response and forwarded across services β€” carried by pure ASGI middleware so the context propagates reliably.

πŸ›‘οΈ Hardened Delivery Path

A nine-job CI pipeline runs Ruff, gofmt, go vet, go test, mypy, pytest, Bandit, Trivy, pip-audit and govulncheck, plus Docker build, Prometheus and Grafana provisioning validation. Third-party actions are pinned to immutable commit SHAs.

πŸ”’ Container Hardening

A reusable Compose anchor applies read-only root filesystems, all Linux capabilities dropped, no-new-privileges, tmpfs /tmp, CPU and memory limits and a graceful stop period. Services run non-root; the Go daemon ships distroless.

🧭 Go Network Daemon

An independent vantage point written in Go that probes DNS, TCP and HTTP reachability and latency, so "the network is broken" can be told apart from "the app is broken".

πŸ““ Runbooks, ADRs & Postmortems

Seven operator runbooks, six architecture decision records and three game-day postmortems β€” each postmortem carrying the exact command that reproduces the incident.

βœ… Tested

59 Python tests and 19 Go tests, at 93.9% line and 85.5% branch coverage on the shared platform library.

How It Was Built πŸ—ΊοΈ

Phase 1

Instrumented Service

A FastAPI service exporting RED metrics β€” rate, errors and duration β€” to Prometheus

Phase 2

Prometheus + Grafana

Scrape configuration, recording rules, SLOs and error budgets, and four provisioned dashboards

Phase 3

Multi-Service Architecture

api, auth and worker on one shared platform library, with circuit breakers and rate limiting

Phase 4

Chaos Engineering

Bounded failure injection through a REST control plane, with the abort path held exempt

Phase 5

Incident Documentation

Game-day postmortems with timelines, root cause and a command to reproduce each incident

Phase 6

Hardening & CI

Container hardening, security scanning, dependency auditing and config validation in CI

Technology Stack

Python FastAPI Go Prometheus Grafana Docker Docker Compose GitHub Actions pytest

Read the Code

Every design decision is recorded as an architecture decision record, every alert links to a runbook, and every incident has a postmortem carrying the command that reproduces it.

View on GitHub β†’