When we set up monitoring in September, the client asked about building a big-screen dashboard collecting everything we could. We didn't. We connected 5 numbers first.
Why Only 5
Too many metrics means no metrics. When something actually breaks, people only look at the few that answer "do I need to get up?" right away:
- API P95 response time. Not the average — averages hide the long tail.
- 5xx ratio, computed per minute, alerting only above 1%.
- MySQL connection count. We've been maxed out at max_connections before; nothing is more direct.
- Redis hit rate. Below 80%, check whether the cache is being bypassed again.
- Message queue depth. Rising and not falling means a consumer is stuck.
Prometheus scrapes every 15 seconds. Grafana is a single screen: 5 panels in a row. Each panel's top-right corner carries one small line: what counts as abnormal, and who to call first. Without that line, nobody dares act on the numbers in front of them.
1. Alerts Go to One Channel
The first version split alerts across 3 channels by metric — then three alerts fired at midnight, all caused by the same slow database. We moved everything into one channel, each message carrying the metric name and current value, so one glance tells you whether it's the same incident.
2. Thresholds Took Two Rounds
5xx started at 0.5% and fired 7 times in the first two days — mostly stray errors from a single user. Moving to 1% sustained for 2 minutes quieted things down. Thresholds aren't guessed; they were set after a week of watching real curves.
3. Don't Let Monitoring Slow the Database Itself
One exporter ran SHOW GLOBAL STATUS every 5 seconds by default; that MySQL was already strained and got slower under it. We changed it to 15 seconds and gave it its own read-only account.
4. The Sixth Metric Was Added, Not Planned
A month in, we added "disk space remaining". The trigger: backup files filled a drive — all 5 original metrics green, yet pages stopped accepting image uploads. While we were at it, we set a 7-day retention cleanup on the backup directory.
Now
The screen still holds those 5 plus 1; nothing piled on since. The on-call sheet gained a line: check the Grafana screen first, then the logs.
Once a new hire asked why we don't build one of those rotating big-screen dashboards. I said: the day it's the first thing you want to look at during an incident, we'll talk.
