Hello to everyone. I have encountered my first security breach! Quite exciting.
I received a few polite emails from abuse networks indicating my outward-facing web & ftp (no mail) server was sending them unfriendly traffic. A quick htop showed that root had executed "pnscan" but I never recalled installing it. I sent it a kill -9, and sure enough, dpkg -s pnscan tells me it's never been installed.
So it seems pretty clear that someone's gotten access to my system in some fashion. The problem is, I don't know where to begin looking. I've scoured logs - I don't see anything obviously fishy there. I've checked bash history - nothing there (though that's easy to circumvent).
Where else should I be looking, or what has my inexperience caused me to miss in the logs? Is this definitely someone who's rooted my server? Could be an FTP vulnerability?
---------- Post updated at 09:23 AM ---------- Previous update was at 09:14 AM ----------
Oops. This should have been posted in Security. I won't repost, I'll let a moderator move it. Sorry for the inconvenience.
That's my worry. But it had been running for 31 hours. So perhaps it was some stupid mistake I made in running some script? I mean, I am fairly new, but I don't think I'm THAT incompetent.
There seem to be no other issues or problems with the system. If someone does have root, why would they not do something more useful or drastic? That's probably useless to speculate upon.
In any case, all I've done is a password change. I'm worried I'm leaving other avenues unexplored, though.
Invaders don't want you to know your system's been compromised, so try to interfere with it as little as possible while still using it for their own purposes.
It's possible they replaced a few programs for their own purposes to resist attempts at removal(something known as 'rootkitting'). You should try something like rkhunter to see if it finds any known rootkits on your system.
While it seems to be a reflex both new and seasoned Linux admins fall for and while information can be gleaned from existing files, killing processes without recording details first does not help or help speed up the fact-finding process as clues like deleted files on open file descriptors and environment information like user details, working directory and connection data is lost.
A second thing, and that may be just me favoring cold, hard data over an account of things any day, is that it is more efficient to tell us what terms exactly you have looked for and in which log files as "anything obviously fishy" doesn't convey much. More importantly, if you never have experienced a breach of security then the best thing to do is do nothing. Take a step back, ask for advice and read. While old and decommissioned the CERT Intruder Detection Checklist still can provide you with aspects of your system to check. Finally I would not install software but assess the system and perform log analysis first. List which 'net-facing software and which versions are installed including applications you run on top of the web server and including any plugins those applications use. Wrt logs: if you have a separate known safe workstation (hell, it could even be a virtualized guest on a home machine) then I suggest you pull in utmp, wtmp, btmp, lastlog, the system and daemon logs and run Logwatch on it. Easiest, quickest way IMO to generate leads.
A rogue process running as root being as good as any other clue I'm more interested in how this happened.
Apart from changing all passwords do consider the system compromised until a conclusion tells you otherwise. Best stop or restrict access to any 'net-facing service that are not vital in the fact-finding phase (meaning that if the machine is not local you'll only want SSH access).
While I would love to leave a port scanner running on my system while I gather details ineptly, I must disagree with the generalization of this statement. My first priority is to stop whatever malicious activity may be occurring on my server that may be affecting the well-being of someone else's server. In this case, my regard for other system administrators trumps my love of data.
Again, when AT&T, abuse networks and other sysadmins are emailing me, this is actually the opposite of what anyone should do.
I agree. Data trumps anecdotes. However, I'm not asking anyone else to diagnose the problem. That statement was merely an indication that the log files aren't flashing "WARNING: INTRUDER" type messages. I was hoping someone might suggest which logs were most likely to contain information, and what this type of problem might look like in them.
Your suggestion about utmp, wtmp, lastlog, etc is sound, and that will certainly be a step I take.
The last command revealed two logins without IPs under my personal login. Perhaps that's meaningless, but the limited number of places I log in from all have IPs recorded.
I also realized that this production server had many settings cloned from a development server. Which means that non-root user had sudo access, and ssh was accepting passwords and PAM.
I have since switched SSH to key auth only, completely removed any and all non-system users from sudo-enabled groups, as well as revisited my iptables firewall. I haven't been able to correctly limit the OUTPUT chain without killing web services, but I'll keep researching.
At this point, I have seen no other logins, no rogue processes and the victims have reported the port scanning as ceased. That's enough for a tentative declaration of "fixed" while I dig deeper.
---------- Post updated at 03:24 PM ---------- Previous update was at 09:49 AM ----------
Here's another interesting development. I have found that the system looks to be sending out requests that computers all over the internal network answer on port 8080. When I plug the network cable in, the flood begins. When I unplug, it stops.
When I moved all functionality to another server, and booted into a LiveCD to reinstall the OS from scratch? It's still doing it. Plug network in, traffic surge. Unplug, traffic stops.
I'm in the process of capturing the outbound data (only had the inbound answer) to get more info. But it seems that whatever this infection is, it runs at boot time. Has anyone ever experienced something like this?