SRE Playground is a planned engineering lab for studying what happens after a system has been deployed.
It is not intended to be a typical product application or another infrastructure provisioning demo. The project is designed around Day 2 operations: the incidents, maintenance work, recovery decisions, and operational uncertainty that begin once real services are already running.
Why Day 2 incidents
Day 1 work gets a service into production. Day 2 work keeps it useful when traffic changes, dependencies slow down, certificates expire, deployments fail, disks fill, alerts become noisy, or an operator has to recover a system under time pressure.
The project will turn those situations into repeatable scenarios that can be triggered, observed, diagnosed, recovered, and reviewed without affecting a real production environment.
Planned operating model
The current design separates the lab into five responsibilities:
| Layer | Planned responsibility |
|---|---|
| Environment | Provision isolated, short-lived infrastructure for each exercise |
| Workload | Run small services with intentional and documented failure modes |
| Signals | Collect logs, metrics, health checks, alerts, and deployment events |
| Incident control | Trigger scenarios and record the response timeline |
| Recovery | Execute runbooks, verify service health, and remove lab resources safely |
Terraform is planned as the environment definition, with Docker-based workloads running on Linux infrastructure in AWS. Bash utilities will coordinate scenario setup and teardown. The final observability stack will be selected during the first implementation phase rather than presented as already complete.
Planned incident catalogue
The lab is intended to grow around incidents that resemble everyday IT operations:
- a deployment that passes startup checks but causes elevated errors;
- CPU, memory, disk, or connection-pool saturation;
- a slow or unavailable downstream dependency;
- unhealthy instances remaining behind a load balancer;
- expired credentials, certificates, or configuration;
- backup and restore verification;
- alert storms, missing alerts, and unclear ownership;
- orphaned cloud resources and unexpected cost exposure;
- partial recovery where one signal looks healthy before the user journey is restored.
Each scenario should define the expected symptoms, available evidence, recovery objective, rollback path, and cleanup requirements before it is automated.
Incident lifecycle
provision
-> establish healthy baseline
-> inject one controlled failure
-> detect and triage
-> follow or improve the runbook
-> verify user-facing recovery
-> review the timeline
-> reconcile and destroy resources
The review stage is as important as recovery. A scenario should leave behind a compact incident record: what changed, which signals were useful, what delayed diagnosis, whether the runbook was accurate, and what should be automated next.
What makes the project different
Most portfolio infrastructure projects focus on creating resources. SRE Playground is planned around operating them when assumptions fail.
The goal is not to demonstrate that a healthy architecture diagram can be deployed. It is to practise the judgement required when several signals disagree, the first fix does not work, recovery introduces new risk, or the safest decision is to roll back.
Delivery phases
Phase 1 — One service, one failure
Build the smallest disposable environment, establish a healthy baseline, inject a single resource-saturation incident, and prove that teardown is reliable.
Phase 2 — Signals and runbooks
Add logs, metrics, alerting, response notes, and explicit recovery verification. Compare what the runbook expected with what the operator actually needed.
Phase 3 — Distributed failure
Introduce dependencies, load balancing, failed deployments, partial availability, and incidents where infrastructure health does not match user-facing health.
Phase 4 — Reconciliation and review
Automate resource reconciliation, capture incident timelines, and create a repeatable post-incident review format.
Success criteria
The project will be considered useful when it can:
- reproduce an incident consistently;
- expose enough evidence for diagnosis without revealing the answer immediately;
- measure recovery against a stated objective;
- verify the user journey after the technical fix;
- clean up every resource even when an exercise stops early; and
- produce lessons that improve the next response.
Until those foundations are implemented and tested, the project remains clearly marked as planned.