Setting alert thresholds is a balance between two costs. Alert too late and a warning becomes downtime. Alert too eagerly and the on-call person learns to ignore notifications, including the one that matters. The second failure mode has a proper name, alert fatigue, and it kills more monitoring setups than any missing feature. This article gives starting numbers per workload type and two techniques that keep false alarms out of your pocket.
What thresholds should each workload start with?
There is no single correct threshold, because “normal” on a web server looks nothing like “normal” on a database or a CI runner. The table below is a safe starting point; after 2-4 weeks of real graphs, tune it to your own baseline:
| Metric | Web / app server | Database | CI runner / batch |
|---|---|---|---|
| CPU | Above 85% for 5 minutes | Above 80% for 5 minutes | Above 95% for 30-60 minutes |
| RAM (by available) | Available below 10% | Available below 5%, watch swap too | Available below 10% |
| Swap | Above 20% used and rising | Sustained swap-in fires immediately | Rarely matters, unless jobs hang |
| Disk (space) | Above 85% | Above 80% | Above 90%, reclaimable via cache |
| Disk (inodes) | Above 85% | Above 85% | Above 85% |
Why does every machine type get its own threshold?
Web and app servers face users directly, so the thing you are really protecting is latency. CPU sustained past roughly 85 percent means requests start queueing and response times grow non-linearly. The remaining 15 percent of headroom is your time to react before users can feel anything.
Databases eat RAM by design: MySQL with its InnoDB buffer pool or PostgreSQL with shared buffers will claim most of the memory, and the kernel turns whatever is “left” into page cache. That is why the used column of free -m always looks maxed out, and that is a good thing. The correct signals are available and swap: available dropping low while swap-in stays sustained means the cache is being pushed to disk and query performance is about to fall off a cliff. Database disk thresholds are also tighter (80 percent) because running out of disk mid-write is a data corruption scenario, not merely downtime.
CI runners and batch machines invert the logic: 100 percent CPU is the definition of working correctly. Configure “CPU above 85%” on a build machine and you have hand-built a false alarm generator. For these machines the signal worth alerting on is duration: CPU pinned at 95-100 percent for 30-60 minutes usually means a hung job or an infinite loop, not a build in progress.
Disk deserves special respect for one property: it never recovers on its own. CPU drops when the work finishes, RAM returns when memory is freed, but disk only climbs until a human cleans it. So a disk alert at 85 percent does not mean “incident soon”; it means “schedule a cleanup this week”. If your disk grows 1 percent per day, an 85 percent threshold buys you 2 weeks of runway; a 95 percent threshold buys you 5 days.
What is hysteresis and why is missing it a disaster?
Picture CPU oscillating between 84 and 86 percent for an hour against your 85 percent threshold. Without hysteresis, your inbox collects “ALERT… RECOVERED… ALERT… RECOVERED” a few dozen times. This is called a flapping alert.
Hysteresis fixes it by splitting the threshold in two:
Fire threshold: CPU > 85%
Recover threshold: CPU < 75%
84% -> 87% : alert fires (crossed 85)
87% -> 81% : still alerting (not yet below 75)
81% -> 86% : nothing new (already in alert state)
86% -> 72% : recovery notice (dropped below 75)The 10-point gap between the two thresholds compresses dozens of noisy notifications into exactly 2 meaningful ones: one when it starts, one when it ends.
How many consecutive samples before you alert?
The second anti-noise technique: never alert on the first breaching sample. A heavy cron job running for 40 seconds can push CPU to 95 percent at the exact moment a sample is taken, then everything returns to normal. Alerting on that is alerting on nothing.
The pragmatic rule: require 3 consecutive samples over the threshold. Detection delay then equals sample count times collection interval:
| Collection interval | 2 consecutive samples | 3 consecutive samples | 5 consecutive samples |
|---|---|---|---|
| 30 seconds | 1 minute | 1.5 minutes | 2.5 minutes |
| 1 minute | 2 minutes | 3 minutes | 5 minutes |
| 5 minutes | 10 minutes | 15 minutes | 25 minutes |
Pick based on what the service can tolerate: an online store should detect within 3 minutes, while a box running internal reports can wait 15 without anyone suffering. Note the bottom row: at a 5-minute interval, demanding 5 samples means you find out almost half an hour later. The sparser the interval, the fewer samples you should require.
What is the fastest way to put these thresholds in place?
If you self-host, both Zabbix and Prometheus support hysteresis and duration conditions, but you configure every rule yourself. If you want it working today, hosted agent-based services ship the hard parts prebuilt: with AgentWatch, the agent collects CPU, RAM and disk on a schedule, you set the threshold and consecutive sample count on the dashboard, and alerts go out via Zalo or email. The Free plan covers 2 servers at a 5-minute interval, enough to start recording your baseline before tuning anything.
One last point, and the most important one: thresholds are not a set-and-forget artifact. Once a month, open the graphs and ask two questions. Did any alert fire last month that you ignored without consequence? Loosen or delete it. Did any incident happen that no alert predicted? Add a threshold on that exact metric. A good alerting setup is one where every single notification is worth interrupting your work for.