An alert that pages you when nothing is broken

2026-09-11

monitoringalerts

My monitoring had a check on the resolvers the router uses. There are several of them on purpose: if one is slow or unreachable, the others answer and nothing upstairs notices. The check asked each one to resolve a name and reported a failure if any of them didn't.

For months it was quiet. Then one of the providers started missing about one query in ten, and my phone started buzzing five or six times a day with a red notification that resolved itself two minutes later. Every single one of those alerts was true and every single one was useless: the service it was watching never stopped working for a second, because that is what the other resolvers were for.

An alert like that costs more than it gives. Not because of the noise itself, but because of what the noise teaches you: within a week I was swiping the notification away before reading it, which means the day one of these is real, I will swipe that one away too.

Measure before you tune

The temptation is to bump a threshold until the noise stops. Better to spend ten minutes learning what normal looks like. I ran the same lookup sixty times against each member:

That table answers the question the alert could not. A healthy member does not miss at all, so a miss is a real signal and not a measurement artefact. The bad one misses roughly one try in ten and is also four times slower, which reads like a struggling connection rather than an outage. And, crucially, a second attempt almost always succeeds — so a check that retries twice will mostly be quiet, and will still fire a few times a week for nothing.

What the alert should answer

The question is not "did something fail". It is "do I need to get up". For a redundant set, that means two specific situations and nothing else:

• More than one member failing at once. One member is redundancy doing its job; two at once is not a coincidence — it is the network, the uplink, or something local to the box.
• The same member failing twice in a row. A blip does not repeat itself on a schedule. If member D fails this run and the next, D is genuinely gone, and I have lost a spare I paid for.

Everything else goes into the check's message, where I can read it if I go looking, and out of the notification.

Two runs in a row needs one file

The only state this needs is "who failed last time", and a file in /tmp is enough — if the box reboots, the worst case is one missed correlation:

prev=$(cat "$STATE" 2>/dev/null)
printf '%s\n' "$bad" > "$STATE"
for b in $bad; do
  case " $prev " in *" $b "*) repeat=1 ;; esac
done

The whole change was about fifteen lines of shell. Since it went in, the check has been green, the message has twice said flap: after a hiccup I never had to know about in the moment, and I have not swiped away a single notification.

The rule I took away

Write the alert against the consequence, not the event. "A resolver missed a query" is an event, and events are what a graph is for. "The set no longer has a spare" is a consequence, and consequences are what a notification is for. If you can't say what you would do when the phone buzzes, the check does not deserve a notification channel — it deserves a line in a log you read when you're curious.