Over two weeks ago this site experienced a severe and prolonged slowdown on the legacy Ubuntu 18 server that hosted both the Discourse community.unix.com forum and the Rails man-pages application. The system became practically unusable: page loads stalled, backend processes choked, and even SSH sessions lagged and frequently timed out.
At first, the timing suggested that an Ubuntu 18 automatic unattended upgrade might have damaged the TCP/IP stack. Kernel updates, networking changes, or altered connection-handling behavior were plausible suspects—but this was never confirmed.
What was confirmed was a massive distributed DDoS attack originating from a very large Russian botnet. The attack didn’t appear as a single high-volume surge. Instead, it presented as a constant, relentless stream of short requests: thousands of rotating IPs hammering the server with garbage paths, recursive bot crawls, malformed requests, and rapid-fire HEAD/GET probes. The cumulative effect saturated Apache workers, overflowed the prefork scoreboard, and choked the reverse-proxy path to both the main Discourse application and the Rails man-page service.
Fail2ban offered no effective relief because the attacking IPs rotated faster than jails could meaningfully respond. With such distribution, bans were essentially useless. Coupled with the aging Ubuntu 18 network stack and the legacy Apache2 prefork MPM, the result was a cascading performance collapse across all services.
After migrating the workloads to two new Ubuntu servers—using an updated php-fpm stack, the Apache2 event MPM, and a cleaner TCP pipeline—the same hostile traffic patterns no longer caused any issues. Performance immediately returned to normal.
In short:
- Suspected: Ubuntu 18 unattended upgrade damaging the TCP/IP stack (never proven).
- Confirmed: Large-scale Russian botnet DDoS causing worker saturation, SSL congestion, and proxy failures.
- Solution: Migration to modern Ubuntu servers with updated stacks, Apache2 event MPM, php-fpm, and a refreshed TCP/IP environment.
The issue is now fully resolved.
Thank you for your patience during the botnet attack and server / app migrations and full stack upgrades.