INC-001SEV-2
Slow Checkout
Checkout has become very slow. Nothing is failing, everything just takes seconds.
Symptom
Support is getting complaints: clicking "Place order" spins for three to four seconds before the confirmation appears. Orders do complete. Nobody has deployed anything today.
SLO: 99% of checkouts under 500 ms. SLI: Share of checkout requests that complete in under 500 ms.
Run the incident
Runs on your machine
Running an incident reads real telemetry from the lab running on your computer: traces from Tempo, metrics from Prometheus and logs from Loki. This public page has no lab behind it, and it will not show invented numbers in their place.
git clone https://github.com/jars-demo/otel-demo.git cd otel-demo docker compose up -d --build --wait
Then open http://localhost:3400: the same site, with every live view connected. New here? Start with Chapter 01.
Investigate
Open the service map. Which services are degraded, and which are healthy?
Compare p95 latency per service against the baseline the lab recorded.
In the Trace Explorer, filter checkout traces slower than 2000 ms and open one.
Follow the critical path down the span tree. Which spans account for the time?
Inside inventory-service, is the time spent in its own code or waiting on something?
Open the logs for this trace. Do any errors explain it?
What is the root cause of the slow checkouts?
The full write-up appears once you have answered.