Should I reboot the server as soon as it goes down?

Not as a first move. A reboot wipes RAM state and much of the evidence you need to find the cause, and if the real problem is a full disk or broken config, the service will still be dead afterwards. Reboot only when SSH fails and you have already checked the provider console.

What if I cannot SSH into the server at all?

Use the out-of-band console from your VPS or cloud provider. Almost every panel offers VNC or a serial console, which shows kernel error output and lets you log in without networking. Only if the console is dead too should you consider a hard reset.

How do I find out the server is down before customers do?

Combine an external uptime check that calls your website on a short interval with an agent measuring resources from inside. When a check fails or a metric crosses a threshold, the system pings you on a channel you actually read, such as Zalo or email, usually minutes to hours ahead of the first customer complaint.

What is the OOM killer and what does it have to do with outages?

When RAM runs out, the Linux kernel picks a memory-hungry process and force-kills it. The victim is often your database or main application. The evidence lives in the kernel log: run dmesg or journalctl -k and search for the phrase "Out of memory".

Is there anything left to do after service is restored?

Yes. Spend 15 minutes recording the timeline, symptoms, root cause and fix in a short document. Then set an alert on the exact metric that caused the incident, for example an 85 percent threshold after a full-disk outage, so next time you know before it breaks instead of after.

Server down? A 7-step incident checklist that works

Nobody thinks clearly when a server dies at 2 a.m. That is exactly why you want a written procedure: follow the order, one goal per step, skip nothing. The 7 steps below have survived plenty of real incidents, and every command is ready to paste into a terminal.

Step 1: how big is the incident?

Before touching SSH, spend 60 seconds establishing scope. Treating the wrong problem is worse than treating it late.

# Does the site still answer, and how slowly?
curl -sv -o /dev/null -w 'HTTP %{http_code} in %{time_total}s\n' https://example.com

# Is the server reachable at the network layer?
ping -c 4 203.0.113.10

Read the results against this table:

Symptom	Most likely cause
HTTP 502 or 503, ping still fine	Server alive, application or backend dead
Full timeout, ping lost	Network-level failure, machine-wide resource exhaustion, or a provider incident
HTTP 200 but customers still report errors	Localized issue: DNS, CDN, or only one group of users
Unusually slow but still responding	CPU, RAM or I/O saturation, not dead yet

Also check the status pages of your VPS provider and CDN. If they are having a regional incident, your job is to inform customers, not to fix the server.

Step 2: can you still get in?

ssh -o ConnectTimeout=10 ops@203.0.113.10

SSH works: go to step 3. SSH refused or hanging: open the provider console (VNC or serial console in the control panel). The console shows kernel panic output if there is any, and lets you log in without networking. Only when the console is useless too should you consider a hard reset, because a reboot destroys most of the diagnostic evidence.

Step 3: has a resource run out?

The three usual suspects behind a server that “just died” are a full disk, exhausted RAM and runaway load. Check all three in one minute:

df -h            # disk: is any Use% column at 100%?
df -i            # inodes: full inodes fail writes even when df -h shows space
free -m          # RAM: how many MB left in the available column?
uptime           # load average versus core count (nproc)
dmesg -T | grep -i 'out of memory'   # did the OOM killer just shoot something?

A disk at 100 percent is the number one reason databases and applications die without a sound. Find the space hog fast:

du -xh / --max-depth=2 2>/dev/null | sort -rh | head -15
journalctl --disk-usage    # system logs are a frequent culprit

Need space urgently? journalctl --vacuum-size=200M shrinks the system journal, then clear temp files and package caches. Do not delete files a process still holds open: the space only returns when the process closes them.

Step 4: which service is actually dead?

systemctl --failed                 # which units are in failed state?
systemctl status nginx mysql       # detailed status per service
docker ps -a                       # any container Exited or Restarting?
ss -tlnp                           # which ports are really listening?

ss -tlnp answers the question that matters: is the process listening on the right port. An app that systemctl calls “running” but that is not listening on 443 is, to your users, dead. For containers, watch the STATUS column: Restarting (1) 30 seconds ago means a crash loop, and docker logs --tail 100 <name> will tell you why.

Step 5: what do the logs say?

Never guess when you can read. journalctl is your best friend right now:

journalctl -u nginx --since '30 min ago' -p err   # nginx errors, last half hour
journalctl -k --since '1 hour ago'                # kernel log: OOM, disk, network
journalctl --since '10 min ago' | tail -100       # the last few minutes, all units

What you are hunting for is the first abnormal event, not the last one. Incident chains run long: disk full at 01:47, MySQL write errors at 01:52, the app loses its database connection at 01:53, a customer complains at 02:10. Fix the 01:53 symptom and you will be awake again tomorrow at 2 a.m.

Step 6: how do you restore service safely?

Once the cause is known, escalate gently:

systemctl restart nginx        # restart exactly the broken service
systemctl status nginx         # confirm it actually came up
docker restart api             # same idea for containers
curl -s -o /dev/null -w '%{http_code}\n' https://example.com   # final check from outside

Two rules. Fix the cause before restarting: restarting MySQL on a disk with zero bytes free just kills it again. And change one thing at a time, so you know exactly what worked. Restarting every service at once may save tonight, but it leaves a mystery for next month.

Step 7: what stops this from happening again?

The incident is not over when the website loads. Spend 15 minutes while memory is fresh:

Write the timeline: when it was detected, the root cause, the fix, the total duration. Three months from now, you will thank yourself.
Patch the root: disk filled by logs means configuring logrotate; RAM leaking means scheduling controlled restarts and telling the developers.
Set an alert on the exact metric that just burned you. Every incident telegraphs itself: a disk does not jump from 60 to 100 percent in a minute. If you have no monitoring yet, this is the moment to add it: an agent measuring resources from inside plus an external uptime check turns “customer calls at 2 a.m.” into “Zalo pings you at 5 p.m.” AgentWatch has a free plan covering 2 servers and 10 checks, with alerts over Zalo and email, which fits this exact scenario.

This checklist will not make servers stop crashing. It makes each crash shorter, calmer, and each recovery leaves the system a little harder to knock over than before.