What the decision must mean
A server can be healthy while customer work is stuck. Monitor the outcome the application is supposed to produce alongside technical failures. For each important workflow, identify the starting event, durable intermediate states and evidence of completion. The distance between those states often explains a problem that an error count misses.
Roles and alternatives
Start with a small set of signals people can act on. Availability and response time describe the application; oldest pending work, completion failures and unusual absence of expected events describe the process. Google's SRE monitoring chapter distinguishes symptoms from causes. Use customer-visible symptoms to decide urgency, then technical detail to investigate rather than paging someone for every isolated log entry.
Walk through the change
In an illustrative quote process, the form saves a request and queues a notification. The website returns successfully, but the mail worker stops. Watch the age of the oldest unsent notification, not only web errors. An alert should identify the queue, give a safe inspection command and explain when to retry. Recovery is confirmed when pending work is delivered without losing or duplicating the underlying request.
Check the failure paths
A quiet dashboard may mean no demand, a broken data collector or a stalled entry point. Distinguish those cases. Avoid alerts without an owner or a useful response, and do not put personal messages into alert payloads. Test monitoring by pausing a local worker or simulating a dependency failure, then verify both the alert and its recovery state.
Give each alert a practical response
For the notification example, write the alert around delayed customer work rather than a process name alone. Include the affected queue, the oldest pending item's age, the time the observation was made and a link or command for an authorized operator to inspect it. Keep personal message content out of the alert. The first responder needs enough context to distinguish a stopped worker from a failing mail dependency without receiving every customer's details.
Define what the operator may safely do. A runbook can start by checking worker health, reviewing a sanitized recent failure and inspecting whether work is leased or already completed. It should explain when retry is safe, when an expired lease can be reclaimed and when to stop and escalate. Retrying every item blindly can repeat an external action that completed before the response was lost. Where the external system cannot provide a clear answer, record the uncertainty and reconcile it rather than inventing a clean success state.
Test the entire alert path in a controlled environment. Pause the worker, create a fictitious request and watch the pending state develop. Confirm the alert reaches the intended local test recipient, then restore the worker and verify completion plus recovery notification or status. Test the collector too: if observations stop arriving, a silent graph must not look like a healthy empty queue. Review alert usefulness after an incident. Remove duplicate noise, adjust ownership and improve the first diagnostic step, but do not suppress a symptom merely because it has become familiar. The goal is a person taking the right action on the right work.
An operational review checklist
| Question | Useful evidence |
|---|---|
| Can we distinguish no work from no measurement? | Recent successful observation is visible separately from the count of pending items. |
| Does the alert identify an owner? | A named role knows its response window, inspection route and escalation contact. |
| Is recovery actually complete? | Pending work reaches its intended business state; restarting a process alone does not close the incident. |
| Can retry repeat a side effect? | The runbook checks completion evidence and treats ambiguous external outcomes as reconciliation work rather than automatic failure. |
Agree these rules
What does success mean for this workflow? How long can work wait before someone needs to intervene? Who receives the alert and what can they safely do? A custom-software review should produce a short operational map, not a wall of metrics nobody uses.