INC-004SEV-2
Inventory Service Failure
About one checkout in three fails. Retrying usually works.
Symptom
Customers say checkout "sometimes" fails with an error and works when they try again. The failure rate looks steady, not random spikes.
Run the incident
Runs on your machine
Running an incident reads real telemetry from the lab running on your computer: traces from Tempo, metrics from Prometheus and logs from Loki. This public page has no lab behind it, and it will not show invented numbers in their place.
git clone https://github.com/jars-demo/otel-demo.git cd otel-demo docker compose up -d --build --wait
Then open http://localhost:3400: the same site, with every live view connected. New here? Start with Chapter 01.
Investigate
Which services show an error rate on the service map?
Is the error rate steady or bursty? What share of checkouts fail?
Open an error trace. Which is the deepest span with status ERROR?
Open a successful checkout from the same minute. What differs?
Filter the logs to ERROR for inventory-service. Then click a trace id.
Why do some checkouts fail?
The full write-up appears once you have answered.