INC-003 · Database Slowdown · OpenTelemetry Incident Lab

Symptom

Creating an order now takes around two seconds. Browsing the catalog is as fast as ever. The database team says "the database is fine, CPU is low".

Run the incident

Runs on your machine

Running an incident reads real telemetry from the lab running on your computer: traces from Tempo, metrics from Prometheus and logs from Loki. This public page has no lab behind it, and it will not show invented numbers in their place.

git clone https://github.com/jars-demo/otel-demo.git
cd otel-demo
docker compose up -d --build --wait

Then open http://localhost:3400: the same site, with every live view connected. New here? Start with Chapter 01.

Investigate

1. Check service health
Which services are degraded? Is inventory-service among them?
Open metrics →
2. Inspect metrics
Compare GET /api/products with POST /api/checkout. Which one changed?
Open metrics →
3. Find a slow trace
Open a checkout trace slower than 1500 ms.
Open traces →
4. Inspect the database spans
Look at the INSERT and UPDATE spans. How long does each one take, and how many are there?
Open traces →
5. Read the query
Do the slow queries look expensive? Check db.query.text.
Open traces →
6. Correlate logs
Are there errors in the logs?
Open logs →

What makes order creation slow?

The INSERT statements are missing an indexEvery round trip to PostgreSQL takes about 300 ms longer than it shouldorder-service is CPU boundRedis is slow

The full write-up appears once you have answered.