How six abandoned build daemons took a server down four times in 27 hours
A shared server kept crashing and rebooting itself. The cause wasn't traffic or a bad deploy. It was five forgotten Gradle build daemons quietly eating memory in the background, with no swap to absorb the spike. Here is how we found it and the two fixes that stopped it for good.
One of our shared servers rebooted itself four times in a single day. No deploy went out, no traffic spike, nothing in the change log. Each time, uptime just reset and every site on the box, ours and our clients', dropped for a few seconds. That kind of outage is the worst to debug, because the obvious suspects are all innocent.
Finding the actual cause
The kernel keeps a record of exactly why it kills processes to free memory, so we started there instead of guessing:
journalctl -k -b -1 | grep oom-killerThat log named the real culprit: six separate Java processes running at once. Only one was supposed to be there, the production backend service itself, capped at 512MB. The other five were orphaned Gradle build daemons, left over from development sessions on a backend project we've been building on the same box, each one sitting on 500MB to over 1GB of memory that nobody was using anymore.
Gradle keeps a daemon alive after a build so the next one starts faster. Left alone, it expires after three hours of being idle. Three hours is a normal length for a coding session, so in practice the daemons were never idle long enough to expire before another one started. They just kept stacking up across days.
The second problem made the first one fatal: the server had no swap configured. With swap, a memory spike degrades performance for a few seconds while the kernel pages things out. Without it, the kernel has one option left when memory runs out, and that's to start killing processes.
The two fixes
Neither one was complicated, which is usually a sign the underlying problem was simple too:
- Added an 8GB swapfile as a safety net, with
vm.swappiness=10so the kernel only reaches for it under real pressure, not casually. - Capped the Gradle daemons at the source, in
gradle.properties: idle timeout dropped from 3 hours to 15 minutes, and daemon heap capped instead of left unbounded.
We also changed a habit on our side: running ./gradlew --stop at the end of a work session instead of trusting the timeout to clean up after us. On a server with headroom to spare that habit wouldn't matter. On a server running close to its memory ceiling by design, because that's how you keep hosting affordable, it's the difference between a quiet afternoon and four reboots.
Why this is worth writing up
Nothing about this was exotic. No corrupted database, no bad code, no attack. It was ordinary developer tooling doing exactly what it was built to do, on a server that couldn't absorb the accumulation. The fix wasn't a clever patch, it was a swapfile and a timeout value, plus a habit change on our end. If you run any kind of daemon-based build tool (Gradle, but also things like esbuild's watch mode or long-lived language servers) on a shared or memory-constrained box, it's worth checking free -h and your process list today rather than after the third unexplained reboot.
If your own infrastructure has been rebooting for reasons nobody can quite explain, talk to us, this is the exact kind of problem we like digging into.