/var/www/xnett.org/linux/things-ive-learned-managing-linux-infrastructure.md
[LINUX]Things I've learned managing Linux infrastructure
Practical lessons, troubleshooting notes and solutions collected from real-world Linux environments.
- author:
- Rolf
- date:
- 2026-09-07 23:54
- read:
- ~2 min
Some lessons only really land after they've cost you an evening. Here are a handful that stuck.
systemd's restart limits are a feature, not a bug
A service crash-looping every few seconds looks like a systemd problem the first time you see it — the unit refuses to restart at all after a while. It's actually StartLimitIntervalSec and StartLimitBurst doing exactly what they're supposed to: stopping a broken service from hammering the system forever.
systemctl status myservice
# look for: "start request repeated too quickly"
journalctl -u myservice --since "10 min ago"The fix is never systemctl reset-failed on a loop — that's treating the symptom. Find out why it's crashing first.
Disk space fails silently until it doesn't
A filesystem sitting at 82% used feels safe right up until a log file, a temp directory, or a stray core dump pushes it to 100% — at which point the failure mode is often much worse than "disk full," because whatever was writing to it (a database, a message queue) can behave unpredictably when writes start failing.
| Signal | Tool | Why it matters |
|---|---|---|
| Filesystem usage trend | df -h, node-exporter |
Catches slow leaks before they're urgent |
| Largest consumers | du -ah . | sort -rh | head |
Finds the actual offender fast |
| inode exhaustion | df -i |
A full filesystem can happen with plenty of bytes free |
SELinux denials are data, not obstacles
"Just disable SELinux" is the fastest way to trade a debuggable problem for an undebuggable one later. audit2why and audit2allow turn a denial into either a specific context fix or a targeted policy module — both far better than turning the whole subsystem off.
ausearch -m avc -ts recent | audit2whyAutomate the boring, repeatable parts — deliberately
Ansible earns its keep the moment you have more than one server doing the same job. But automation cuts both ways: an unattended weekly patch job needs a backup-first, verify-after, and a clear status trail, or it just becomes a faster way to break several machines at once instead of one.
The goal isn't "no manual work." It's "manual work only where a human's judgment actually adds something."
None of this is advanced. It's mostly paying attention to the boring signals before they become incidents — which, in infrastructure work, is most of the job.
# EOF