INC-999 · Final Incident Challenge · OpenTelemetry Incident Lab

Symptom

Checkout latency has increased. Some checkouts fail with HTTP 502. Payment looks healthy. Inventory looks healthy. Database query times look normal. Nobody deployed anything. Find the root cause.

Run the incident

Runs on your machine

Running an incident reads real telemetry from the lab running on your computer: traces from Tempo, metrics from Prometheus and logs from Loki. This public page has no lab behind it, and it will not show invented numbers in their place.

git clone https://github.com/jars-demo/otel-demo.git
cd otel-demo
docker compose up -d --build --wait

Then open http://localhost:3400: the same site, with every live view connected. New here? Start with Chapter 01.

Investigate

1. Check service health
Which services are degraded? Which are healthy?
Open metrics →
2. Inspect metrics
Payment, inventory and the database are healthy. So where could order-service be spending its time?
Open metrics →
3. Find a slow trace
Open a checkout trace slower than 1000 ms and read the order-service spans carefully.
Open traces →
4. Explain the gap
Where is the gap? What does order-service do at the start of a checkout, before its first child span?
Open traces →
5. Check the connection pool
Look at the connection pool metrics for order-service.
Open metrics →
6. Correlate logs
What do the failed checkouts say?
Open logs →

What is the root cause?

Redis latencyPostgreSQL is sloworder-service holds a pooled database connection for the whole checkout, including the inventory and payment calls, so the small pool runs out and requests queue for a connectionA downstream service times out

The full write-up appears once you have answered.