Skip to content
Incident LabOpenTelemetry
← Incident Lab

INC-002SEV-1

Payment Failure

Every checkout fails with HTTP 502. Browsing products still works.

Symptom

The storefront shows "Something went wrong" on every checkout attempt, and the API returns HTTP 502 Bad Gateway. Product pages load fine. Revenue has dropped to zero.

Run the incident

Runs on your machine

Running an incident reads real telemetry from the lab running on your computer: traces from Tempo, metrics from Prometheus and logs from Loki. This public page has no lab behind it, and it will not show invented numbers in their place.

git clone https://github.com/jars-demo/otel-demo.git
cd otel-demo
docker compose up -d --build --wait

Then open http://localhost:3400: the same site, with every live view connected. New here? Start with Chapter 01.

Investigate

  1. Which services show errors on the service map?

  2. Is the whole API failing, or only one operation?

  3. In the Trace Explorer, turn on "Errors only" and open a checkout trace.

  4. Which is the deepest span with status ERROR? What does its exception event say?

  5. Open the logs for the trace. What did each service report?

    Open logs →
  6. Did the failed checkout leave stock reserved?

Why does checkout return 502?

Choose the root cause

The full write-up appears once you have answered.