OpenTelemetry Incident Lab
Break it.
Trace it.
Find it.
Fix it.
A hands-on observability lab where you intentionally break a distributed application and use OpenTelemetry traces, metrics, and logs to find the root cause.
- OpenTelemetry
- Collector
- Grafana
- Tempo
- Prometheus
- Loki
- POST /api/checkout api-gateway
- checkout order-service
- order.validate order-service
- HGETALL GET redis
- order.reserve_inventory order-service
- EVALSHA redis
- order.charge_payment order-service
How the lab works
- 01
Break it
Inject a controlled fault: Redis latency, a failing payment provider, a slow database, an exhausted connection pool.
- 02
Trace it
Follow requests through every service. Metrics tell you something changed; traces show where.
- 03
Find it
Separate self time from waiting, follow the critical path, read the exception, correlate the logs.
- 04
Fix it
Apply the fix, then prove recovery against the baseline you measured before anything broke.
Live System
Live System
When you run the lab, this panel shows each service's health, request rate, error rate and p95 latency, read from Prometheus every five seconds. This public page has no lab behind it, so it shows nothing rather than invented numbers.
docker compose up -d --build --wait
A small system you can hold in your head. Four FastAPI services, PostgreSQL and Redis. A checkout crosses all of them, so one click produces a trace of about 25 spans.
A real telemetry pipeline. Every service sends traces, metrics and logs over OTLP to an OpenTelemetry Collector, which routes them to Tempo, Prometheus and Loki. Grafana and this site read from them.
Nothing faked. Faults are real: latency is added on the network path, errors are raised in code. Every live number comes from the backends.
See the architecture →Three signals, three questions
Incidents
Open the Incident Lab →- INC-001Slow CheckoutCheckout has become very slow. Nothing is failing, everything just takes seconds.
- INC-002Payment FailureEvery checkout fails with HTTP 502. Browsing products still works.
- INC-003Database SlowdownPlacing an order got slow. Product browsing is still fast.
- INC-004Inventory Service FailureAbout one checkout in three fails. Retrying usually works.
- INC-005Cascading LatencyThree services report high latency at the same time. Only one of them is the cause.
- INC-999Final Incident ChallengeCheckout latency is up and some checkouts fail. Every dependency looks healthy.
The workshop
16 chapters, about 5 hours, from "what is observability?" to a final incident challenge.
Every command says what it does, why, and what you should see. Every chapter ends with a challenge.
An educational project inspired by real observability architectures. It is not a production observability platform and it is not the official OpenTelemetry Demo.