Incident Lab
Something is broken. How do you know why?
Each incident starts with a symptom, not an answer. Measure the baseline, let the lab break something, then use traces, metrics and logs to find the root cause. Fix it and prove the system recovered.
Incidents
- INC-001SEV-2Slow CheckoutCheckout has become very slow. Nothing is failing, everything just takes seconds.Investigate →
- INC-002SEV-1Payment FailureEvery checkout fails with HTTP 502. Browsing products still works.Investigate →
- INC-003SEV-2Database SlowdownPlacing an order got slow. Product browsing is still fast.Investigate →
- INC-004SEV-2Inventory Service FailureAbout one checkout in three fails. Retrying usually works.Investigate →
- INC-005SEV-2Cascading LatencyThree services report high latency at the same time. Only one of them is the cause.Investigate →
- INC-999SEV-2Final challengeFinal Incident ChallengeCheckout latency is up and some checkouts fail. Every dependency looks healthy.Investigate →
Incident Control
Break things by hand: turn on a fault, send traffic, and watch the telemetry change. Running an incident above does the same for you, without telling you which fault it chose.
Runs on your machine
Incident Control reads real telemetry from the lab running on your computer: traces from Tempo, metrics from Prometheus and logs from Loki. This public page has no lab behind it, and it will not show invented numbers in their place.
git clone https://github.com/jars-demo/otel-demo.git cd otel-demo docker compose up -d --build --wait
Then open http://localhost:3400: the same site, with every live view connected. New here? Start with Chapter 01.