Home · About · Services · Industries · Projects · Products · Process · Blog · FAQ · Contact
Uptime alerts without alert fatigue: what to page on and how How we design uptime and server alerts that people still act on: multi-location checks, symptom-based rules, page versus ticket, Alertmanager grouping and…
The site went down at 2am on a Saturday. The alert fired. It fired in the same channel that had received 40 other alerts that week, most of them a CPU spike that cleared itself within a minute. Nobody looked until a customer emailed on Monday. The monitoring worked. The alerting had trained people to ignore it.
The short answer: uptime alerts stay trusted when every alert that wakes someone is urgent, real and needs a human. Check from more than one location and require repeated failures before alerting. Alert on what users feel, not on every metric. Split urgent pages from next-day tickets, group related alerts, attach a runbook, and delete any alert nobody acts on.
This post sets out how we design alerts for the servers and applications we look after. The examples use Prometheus, the Blackbox exporter and Alertmanager, but the same rules apply to Uptime Kuma, Grafana alerting or a hosted uptime service.
Why people learn to ignore alerts Alert fatigue is not a failing of the on-call person. It is a predictable response to a system that cries wolf. The usual causes:
Single-probe uptime checks. One failed request from one location, often a network blip between the checker and your server, becomes a page. Cause-based alerts. High CPU, high memory or a busy disk fire constantly on healthy servers. A server at 90 per cent CPU that is answering requests quickly is doing its job. No duration. A threshold crossed for ten seconds alerts the same as one crossed for ten minutes, so short spikes produce a constant stream of fire-and-resolve pairs. One incident, twenty messages. When a database stops, every service that depends on it alerts separately, and the real cause is buried. Everything at the same priority. A certificate expiring in three weeks arrives on the same channel, with the same sound, as the site being down. No owner and no next step. The alert says something is wrong but not who should act or what to do first. Each of these is fixable in configuration. The harder part is agreeing that fewer alerts is the goal.
Decide what deserves a page Before writing rules, sort every possible alert into one of three groups. We write this down with the client, because it decides who gets disturbed and when.
Page: users are affected now, or will be within the hour, and a person can do something about it. The site is down from several locations, the error rate is high, the disk will be full tonight. Ticket: something needs attention within a working day or two, but nothing is broken yet. A certificate expires in 14 days, a backup did not arrive, a replica is lagging. Dashboard only: useful for diagnosis and trends, never sent to anyone. CPU, memory, load average, connection counts. A useful test for any proposed page: if this fired at 3am, would the person woken up do something other than acknowledge it and go back to sleep? If not, it is a ticket or a dashboard panel. Coverage hours and response times for pages are agreed in the retainer, so the page list should be short enough that those commitments are realistic.
Uptime checks that do not cry wolf A good uptime check behaves like a user and is patient about the network. In practice that means four things. Check a real endpoint, such as a health route that touches the database, not just the home page served from cache. Run the check from at least two locations outside your own network. Only treat the service as down when most locations agree. And require the failure to last a few minutes before anyone is told.
With the Blackbox exporter, each probe exposes probe_success as 1 or 0. Run a Blackbox exporter in two or more locations, scrape them all from one Prometheus with a location label on each job, and keep instance set to the target URL. This rule then fires only when at least two locations have been failing for three minutes:
groups:
- name: uptime
rules:
- alert: SiteDown
expr: count by (instance) (probe_success{job="blackbox-http"} == 0) >= 2
for: 3m
labels:
severity: page
annotations:
summary: "{{ $labels.instance }} failing from {{ $value }} locations"
runbook: "https://wiki.example.com/runbooks/site-down"
- alert: CertificateExpiringSoon
expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 < 14
for: 1h
labels:
severity: ticket
annotations:
summary: "TLS certificate for {{ $labels.instance }} expires in under 14 days"The for clause is the single most effective noise filter you have. A probe that fails once and recovers never reaches Alertmanager. Three minutes is a reasonable default for a public site; a payment or login endpoint may justify less, an internal tool more. In Uptime Kuma the equivalent settings are retries and the retry interval on each monitor.
The certificate rule is a ticket, not a page. Most ACME clients renew 90-day certificates around 30 days before expiry, so a certificate with under 14 days left means renewal has been failing for two weeks. That needs fixing this week, not at 3am. Revisit the threshold as lifetimes shrink: the CA/Browser Forum maximum fell to 200 days in March 2026 and drops to 100 days in 2027 and 47 days in 2029, so renewal windows and this alert will tighten with them.
Alert on symptoms and predictions, not raw thresholds For server resources, alert on the outcome you are worried about rather than a static number. A disk at 85 per cent that grows by a few megabytes a week is fine. A disk at 60 per cent that is filling at a gigabyte an hour is an incident tonight. Prometheus can project the trend with predict_linear:
- alert: DiskWillFillIn24h
expr: |
predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h], 24 * 3600) < 0
and node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes < 0.2
for: 30m
labels:
severity: pageThe second condition stops a nearly empty disk with a brief burst of writes from paging anyone. The same thinking applies to applications: alert on the share of requests returning 5xx errors or the response time users see, measured over five or ten minutes, rather than on CPU. CPU belongs on the dashboard you open after the symptom alert fires.
Group, route and suppress in Alertmanager Rules decide what is wrong. Alertmanager decides who hears about it, how often and through which channel. Most alert fatigue we see in inherited setups comes from leaving its defaults untouched. This configuration sends pages to an on-call channel, sends tickets only during office hours, groups related alerts into one message and mutes downstream alerts when the site is already known to be down:
route:
receiver: tickets
group_by: ['alertname', 'instance']
group_wait: 30s
group_interval: 5m
repeat_interval: 12h
routes:
- matchers: ['severity="page"']
receiver: oncall
repeat_interval: 1h
- matchers: ['severity="ticket"']
receiver: tickets
active_time_intervals: ['office-hours']
time_intervals:
- name: office-hours
time_intervals:
- weekdays: ['monday:friday']
times:
- start_time: '09:00'
end_time: '18:00'
location: 'Asia/Kolkata'
inhibit_rules:
- source_matchers: ['alertname="SiteDown"']
target_matchers: ['severity="ticket"']
equal: ['instance']
receivers:
- name: oncall
# phone, push or chat integration for the on-call person
- name: tickets
# shared channel or issue trackerA few details matter here. group_wait gives related alerts 30 seconds to arrive together, so a failing database produces one notification listing several services instead of ten separate messages. repeat_interval controls how often an unresolved alert is sent again; a page that repeats every few minutes gets muted by the recipient, which is worse than a sensible hourly reminder. Outside office hours the ticket route is muted; anything still firing is sent once the window opens, and anything that resolved overnight is never sent. The inhibit rule stops lesser alerts for an instance that is already down from adding noise to an incident everyone knows about.
For planned work, silence alerts instead of letting them fire and asking people to ignore them. A silence has an end time and a comment, so it cannot be forgotten in the way that a disabled rule can:
amtool --alertmanager.url=http://localhost:9093 silence add \
instance="app.example.com" \
--duration=2h --comment="Database upgrade, ticket 412"Every alert needs an owner and a runbook An alert that arrives without context costs the recipient ten minutes of working out what it means. We attach a runbook link to every paging rule, as in the annotation above. The runbook is short and written for someone tired: what the alert means, how to confirm it is real, the first two or three safe actions, and who to escalate to if those do not work.
The alert text should also make sense on a phone lock screen. Name the service and the symptom in the summary, such as "api.example.com failing from 3 locations", rather than an internal job name. Send pages to a named person on a rota, with a shared channel as a record. An alert that goes to everyone is read by nobody.
Keep the alert list honest Alert rules decay. Services are retired, thresholds drift, and a rule added during one incident stays forever. We review the alerts that fired each week or fortnight and ask three questions of each one: was it real, did someone act on it, and was the action something a person needed to do. An alert that fires repeatedly and leads to no action gets its cause fixed, its threshold or duration changed, its severity lowered, or it gets deleted.
A rough health check for a small setup: pages should be rare enough that each one is investigated properly, and almost every page should lead to an action. If the on-call person is acknowledging pages without opening anything, the system is training them to ignore it again.
A checklist for alerts people trust Uptime checks hit a real health endpoint from at least two external locations. Down means most locations agree, and the failure has lasted a few minutes. Every rule is labelled page, ticket or dashboard-only, and the page list is short. Pages are for user-facing symptoms and imminent failures, not CPU or memory. Related alerts are grouped, downstream alerts are inhibited, and repeats are spaced out. Tickets wait for office hours; planned work uses silences with an expiry. Every page has a runbook link and goes to a named person on a rota. Fired alerts are reviewed regularly, and any alert nobody acts on is fixed or removed. If your team already ignores the alert channel, start by muting everything except a multi-location uptime check and a disk prediction, then add rules back one at a time as each proves it deserves to interrupt someone.
Infigini IT Services · 11 Kanwali Road, Dehradun 248001, India · info@infigini.com · +91 95684 87789