INC-005SEV-2
Cascading Latency
Three services report high latency at the same time. Only one of them is the cause.
Symptom
Alerts fire for api-gateway, order-service and inventory-service, all for high latency, all at once. Three teams are paged, and each one says "it is not us, we are just waiting on someone else". Redis and PostgreSQL look healthy.
Run the incident
Runs on your machine
Running an incident reads real telemetry from the lab running on your computer: traces from Tempo, metrics from Prometheus and logs from Loki. This public page has no lab behind it, and it will not show invented numbers in their place.
git clone https://github.com/jars-demo/otel-demo.git cd otel-demo docker compose up -d --build --wait
Then open http://localhost:3400: the same site, with every live view connected. New here? Start with Chapter 01.
Investigate
How many services are degraded? Are the data stores healthy?
Which operation slowed down? Is browsing affected?
Open a checkout trace slower than 2000 ms.
Walk down the critical path. Where does the self time sit?
In INC-001 the time was inside a Redis span. Where is it now?
Are there errors?
Which component is responsible for the latency?
The full write-up appears once you have answered.