DEV Community

sunah kim
sunah kim

Posted on Originally published at zenn.dev

The Kernel-Update Reboot Should Take 1 Minute. It Took 10 — the Slow Part Was Human

What happened

Our nightly auto-patching (a systemd timer running dnf upgrade) deliberately does not reboot. On days when the batch includes packages like the kernel or glibc — things that only take effect after a restart — it just logs REBOOT-NEEDED and leaves the reboot to a human.

The 33 piled-up updates from my previous post included exactly that: kernel and glibc. Rebooting the production site follows a simple runbook: put up the maintenance page → reboot → verify → take the page down. The server itself is down for under a minute. Yet the maintenance page was actually visible for about 10 minutes.

The slow part wasn't the server. It was me.

Procedure v1 failed — maintenance ON came too early

My original procedure looked like this:

# v1: maintenance ON → update → reboot
sudo touch /var/www/html/.maintenance    # maintenance page ON (flag file)
sudo dnf upgrade -y --releasever=latest  # ← takes 1–2 minutes here
sudo reboot
Enter fullscreen mode Exit fullscreen mode

Downloading and applying packages takes a minute or two, during which the server can serve traffic just fine — but visitors see a maintenance page. Pointless downtime from the user's perspective. So I swapped the order:

# v2: update → maintenance ON → reboot
sudo dnf upgrade -y --releasever=latest
sudo touch /var/www/html/.maintenance
sudo reboot
Enter fullscreen mode Exit fullscreen mode

Now the only outage window is around the reboot itself — under a minute. In theory.

Measured: 10 minutes — the bottleneck was human

Step Expected Measured
dnf download/apply (before maintenance) 1–2 min ~2 min (no impact)
Maintenance ON → reboot → OS back up ~1 min ~50 s
Human reopens browser SSH and logs back in (not budgeted) ~9 min
Removal command (rm) seconds seconds

Checking uptime -s afterwards showed the reboot itself finished in about 50 seconds. The remaining 9 minutes were me: the browser SSH session drops on reboot, I reopen the tab, retry a few connection errors wondering "is it up yet?", get distracted by something else, and finally log in and type the removal command.

I had designed the procedure to minimize server downtime — but I was measuring the wrong thing. What determines how long the maintenance page stays up isn't the reboot time; it's the time until a human types the next command.

The fix — take the human out of the removal step

Two changes.

(1) Prepare the one-liner to paste after reconnecting

Right before rebooting, put the next command in a note:

sudo rm -f /var/www/html/.maintenance && curl -sk -o /dev/null -w '%{http_code}\n' https://localhost/
Enter fullscreen mode Exit fullscreen mode

Reopen the session, paste, done. No "wait, what do I type after logging in?" time.

(2) A systemd oneshot that clears the maintenance flag after reboot

# /etc/systemd/system/maintenance-clear.service
[Unit]
Description=Clear maintenance flag after reboot
After=httpd.service
Requires=httpd.service

[Service]
Type=oneshot
# Wait until localhost returns 200 before clearing the flag
ExecStartPre=/usr/bin/bash -c 'for i in $(seq 1 30); do curl -sfk -o /dev/null https://localhost/ && exit 0; sleep 2; done; exit 1'
ExecStart=/usr/bin/rm -f /var/www/html/.maintenance

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

The key is the ExecStartPre wait. If you remove the flag unconditionally at boot, the maintenance page disappears before the app is up and visitors see a 502. Verify "healthy means exactly this value" first, then remove. If the health check never passes, the flag stays and the maintenance page remains — the failure mode is safe.

With this in place the human part of the runbook ends at "dnf → maintenance ON → reboot". I only come back later to check uname -r.

Verification

What I checked after the reboot:

uptime -s                    # reboot time (backs up the actual outage duration)
uname -r                     # running on the new kernel?
sudo dnf history info last   # what was applied
Enter fullscreen mode Exit fullscreen mode

On top of that: during maintenance, I confirmed the four main pages returned 200 from inside using a bypass cookie; after removal, I recorded 200s and response times for the same four pages via their external URLs. The recovery time I log is not "when I removed the flag" but when the site looked healthy from outside.

Takeaways

  • Nightly auto-patching doesn't reboot by design. Kernel/glibc days need a human reboot
  • Turn maintenance ON right before the reboot — showing it during the download is pointless exposure
  • Measure downtime not as "time the server was down" but as "time until the human types the next command." Here: 50 seconds vs. 10 minutes
  • Bundle the removal into the reconnect one-liner, or automate it with a health-checked oneshot, and the "human" row disappears from the timetable

Add a "human" row to the time estimates in your runbooks. It's usually the longest one.

Top comments (0)