Relative newbie to Linux so please be kind and assume I've done little in the way of command line but i have been thrusted into this position.
Here goes. There is a perl script on my box that is using me as a mail server. It is contacting other mail servers to the point of slowing down the box. How do I find the script via terminal and how do I then remove it?
thanks for your help.
Hi,
There are a few possible approaches here, but first a bit more info would be ideal. Is this Perl script somewhere in someone's Web space on a shared Web server running Apache, and somehow it's getting triggered and causing the shared Web server to start sending out spam ? Or is the situation something different ?
If you can give some idea of the typical role of this server, what exact OS and distribution it's running, what processes you'd expect to see running on it (i.e. does it ever run any Perl for legitimate reasons), and what your findings are so far, that would be a big help.
Every running program (more precisely: every instance of a running program, because the same program could be started more than once) is a "process" in UNIX. Processes are managed in a table by the kernel and there is a command to display (parts of) this table: ps . ps has many many options (too many to explain them all here) but you might want to start with this (a sample output is below):
# ps -fe
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 Feb10 ? 00:00:16 /usr/lib/systemd/systemd --switched-root --system --de serialize 20
root 2 0 0 Feb10 ? 00:00:00 [kthreadd]
root 3 2 0 Feb10 ? 00:00:16 [ksoftirqd/0]
root 5 2 0 Feb10 ? 00:00:00 [kworker/0:0H]
root 7 2 0 Feb10 ? 00:05:34 [rcu_sched]
root 8 2 0 Feb10 ? 00:00:00 [rcu_bh]
root 9 2 0 Feb10 ? 00:02:48 [rcuos/0]
root 10 2 0 Feb10 ? 00:00:00 [rcuob/0]
root 11 2 0 Feb10 ? 00:00:00 [migration/0]
root 12 2 0 Feb10 ? 00:00:04 [watchdog/0]
root 13 2 0 Feb10 ? 00:00:03 [watchdog/1]
root 14 2 0 Feb10 ? 00:00:00 [migration/1]
root 15 2 0 Feb10 ? 00:00:01 [ksoftirqd/1]
root 17 2 0 Feb10 ? 00:00:00 [kworker/1:0H]
...
What you see is the owner of the process ("UID"), the process' ID ("PID"), which is unique for every running process. (Unique in the sense that every running process is guaranteed to have a different number. Once it stops and the number becomes unused it can be reused by the next process.). Furthermore there is the parent process' ID ("PPID", more on that below) and the command used to invoke the process ("CMD") - some are enclosed in square brackets (i.e. "[kthreadd]"), signifying kernel threads where no real command in the classical sense was used to start them.
Use the grep-utility to filter perl-processes:
# ps -fe | grep perl
bakunin 8842 25710 0 Feb21 pts/0 00:00:00 /opt/bin/perl /foo/bar/myperlprog
bakunin 15520 15518 0 19:57 pts/1 00:00:00 /opt/bin/perl /foo/otherprog
...
To stop a process note its PID and send it a signal (a kind-of message), using the kill command:
# kill -15 PID1 PID2 PID3 ...
15 is the signal which tells a process to stop running and relinquish all its allocated resources: this is the most gentle and preferable way to do it, because a process will not just be stopped no matter what but given the opportunity to i.e. close opened files, release shared memory segments which won't be needed after it stops, etc. - in one word, cleaning up. Well-written programs will honor this signal and indeed quit as soon as they managed to clean up.
Less well-written programs might ignore this, though, and then (but only then!) you can use signal 9 instead. This is not a signal in the common sense (at least not to the program), but the command to the OS kernel to immediately terminate the program, regardless of it wanting to stop or not. If signal 15 is the asking to kindly commit suicide after phrasing your last will, signal 9 is a headshot. Note that signal 9 is used if you must, not because you can! It harms the stability of the kernel to use it and hence more gentle methods are preferred as long as they work.
A word about process hierarchies and the PPID: all process in a UNIX system are organized in a tree: each process can have multiple child processes which in turn can have one or more children of their own and so on. The root of this process tree is "init" (in modern Linux systems "systemd"), which always has PID 1. Every child process has the PID of its parent in the field PPID, so you can reconstruct the (part-)tree from there. Kill the parent and all children will die equally with it. Kill the init-process and you have shut down the whole system immediately (and a good chance to have damaged the system in the way, so don't try that on a system you need, at least not without necessity).
At last a word about how to avoid the program starting again: you need to find out from where it was started in first place. The PPID field might help with this. Common possibilities include:
starting process: each UNIX system has a booting process and there are two general flavors of this: System V and BSD. System V-like systems execute first programs noted in the file /etc/inittab . If this file exists, have a look there.
run levels: BSD- and System V-like systems use so-called "run levels" and execute a series of start-stop-scripts located in /etc/rc.d/rcN where N is a number between 1 and 6. These are directories in which scripts with names starting with "K" (kill) and "S" are located. When a certain run-level is entered all the the S-scripts in that level are executed. When the run-level is switched, all K-scripts of the current runlevel are executed first, then all S-scripts of the new run-level are executed. Have a look there.
cron: UNIX has its own job-scheduler which can be used to repetitively start certain jobs at certain times. For every user there is its own Job list which you can display by switching into this user account and entering the command:
# crontab -l
You can edit this list with the command
# crontab -e
by removing the line with the call to the perl-program once you have found it.
Notice, that most to all these activities need you to gain access to the root user and that all these activities are potentially harmful to your system. If you do not know exactly what you do - DON'T DO IT! Otherwise you are risking your system. It is possible to voluntarily ruin a UNIX system beyond repair as root .
I hope this helps.
bakunin
Here is what support sent me. I'm suppose to do this myself. Yes, someone else got into terminal cause when remotely logged in I saw this last Login: the feb 23 14:59:13 2017 from phrank.aus.us.siteprotect.com
and then this happened:
top - 13:06:42 up 3 days, 16:16, 1 user, load average: 4.28, 7.87, 9.43
Tasks: 212 total, 2 running, 210 sleeping, 0 stopped, 0 zombie
Cpu(s): 15.6%us, 2.5%sy, 2.7%ni, 18.6%id, 60.0%wa, 0.0%hi, 0.7%si, 0.0%st
Mem: 1928704k total, 1875564k used, 53140k free, 17392k buffers
Swap: 4128764k total, 2257872k used, 1870892k free, 131640k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4600 apache 20 0 2898m 1.3g 832 D 0.7 70.5 216:41.32 perl
11541 mysql 30 10 793m 50m 4516 S 0.7 2.7 25:14.14 mysqld
764 petra195 30 10 130m 37m 29m S 2.3 2.0 0:01.20 php-cgi
606 petra195 30 10 130m 37m 29m S 0.0 2.0 0:01.08 php-cgi
425 root 20 0 80596 26m 15m S 0.0 1.4 0:00.51 sw-engine
32159 psaadm 30 10 178m 10m 7432 S 0.0 0.6 0:02.80 sw-engine-fpm
12538 tomcat 20 0 647m 10m 712 S 0.0 0.6 2:38.43 java
710 psaadm 30 10 175m 7428 5132 S 0.0 0.4 0:00.04 sw-engine-fpm
3986 root 20 0 86160 6452 1820 S 0.0 0.3 0:18.78 sw-engine
18798 root 30 10 219m 5568 1532 S 1.0 0.3 20:20.56 fail2ban-server
4552 apache 20 0 9108 4540 988 R 16.5 0.2 201:07.83 perl
31173 apache 30 10 49924 4340 2712 S 0.0 0.2 0:00.19 httpd
4334 apache 20 0 11224 4128 1000 S 9.9 0.2 330:41.40 rnd
31111 apache 30 10 49792 4060 2504 S 0.0 0.2 0:00.08 httpd
4599 apache 20 0 8252 4052 988 S 4.6 0.2 211:32.19 perl
709 root 20 0 12604 3696 2872 S 0.0 0.2 0:00.05 sshd
31050 apache 30 10 50880 3688 2164 S 0.0 0.2 0:01.51 httpd
8391 apache 30 10 55936 3636 2304 S 0.0 0.2 0:11.95 httpd
[root@dedicated ~]# lsof | grep 4600
perl 4600 apache cwd DIR 253,3 4096 2 /
perl 4600 apache rtd DIR 253,3 4096 2 /
perl 4600 apache txt REG 253,3 5392 2364698 /usr/bin/perl
perl 4600 apache mem REG 253,3 17896 2765063 /lib/libdl-2.12.so
perl 4600 apache mem REG 253,3 200092 2765065 /lib/libm-2.12.so
perl 4600 apache mem REG 253,3 131260 2763674 /lib/libpthread-2.12.so
perl 4600 apache mem REG 253,3 1908112 2758628 /lib/libc-2.12.so
perl 4600 apache mem REG 253,3 19864 2626224 /usr/lib/perl5/auto/Socket/Socket.so
perl 4600 apache mem REG 253,3 145272 2755460 /lib/ld-2.12.so
perl 4600 apache mem REG 253,3 38380 2765062 /lib/libcrypt-2.12.so
perl 4600 apache mem REG 253,3 11368 2625445 /usr/lib/perl5/auto/Fcntl/Fcntl.so
perl 4600 apache mem REG 253,3 1461680 2491814 /usr/lib/perl5/CORE/libperl.so
perl 4600 apache mem REG 253,3 103388 2765077 /lib/libresolv-2.12.so
perl 4600 apache mem REG 253,3 105160 2626816 /usr/lib/perl5/auto/POSIX/POSIX.so
perl 4600 apache mem REG 253,3 12792 2765083 /lib/libutil-2.12.so
perl 4600 apache mem REG 253,3 9604 2764968 /lib/libfreebl3.so
perl 4600 apache mem REG 253,3 113912 2765067 /lib/libnsl-2.12.so
perl 4600 apache mem REG 253,3 16484 2625464 /usr/lib/perl5/auto/IO/IO.so
perl 4600 apache mem REG 253,3 18828 2628558 /usr/lib/perl5/auto/File/Glob/Glob.so
perl 4600 apache 0r CHR 1,3 0t0 3975 /dev/null
perl 4600 apache 1w CHR 1,3 0t0 3975 /dev/null
perl 4600 apache 2w CHR 1,3 0t0 3975 /dev/null
perl 4600 apache 3u IPv4 22086043 0t0 UDP *:40988
perl 4600 apache 4u IPv4 22086040 0t0 UDP *:59998
perl 4600 apache 5u IPv4 22086014 0t0 TCP dedicated.soloservices.net:59216->imta-ch2.sys.comcast.net:smtp (CLOSE_WAIT)
perl 4600 apache 6u IPv4 22086050 0t0 TCP dedicated.soloservices.net:55756->mtain-a-atc-b.mx.aol.com:smtp (CLOSE_WAIT)
perl 4600 apache 7u IPv4 22086018 0t0 TCP dedicated.soloservices.net:59220->imta-ch2.sys.comcast.net:smtp (CLOSE_WAIT)
perl 4600 apache 8u IPv4 22086048 0t0 TCP dedicated.soloservices.net:55752->mtain-a-atc-b.mx.aol.com:smtp (CLOSE_WAIT)
perl 4600 apache 9u IPv4 22086019 0t0 TCP dedicated.soloservices.net:59222->imta-ch2.sys.comcast.net:smtp (CLOSE_WAIT)
perl 4600 apache 10u IPv4 22085763 0t0 UDP *:33352
perl 4600 apache 11u sock 0,6 0t0 22084878 can't identify protocol
perl 4600 apache 12u sock 0,6 0t0 22085460 can't identify protocol
perl 4600 apache 13u IPv4 22086021 0t0 TCP dedicated.soloservices.net:40530->imta-po.sys.comcast.net:smtp (CLOSE_WAIT)
perl 4600 apache 14u IPv4 22086015 0t0 TCP dedicated.soloservices.net:59218->imta-ch2.sys.comcast.net:smtp (CLOSE_WAIT)
perl 4600 apache 15u IPv4 22086020 0t0 TCP dedicated.soloservices.net:59224->imta-ch2.sys.comcast.net:smtp (CLOSE_WAIT)
perl 4600 apache 16u IPv4 22086022 0t0 TCP dedicated.soloservices.net:40532->imta-po.sys.comcast.net:smtp (CLOSE_WAIT)
perl 4600 apache 17u IPv4 22086023 0t0 TCP dedicated.soloservices.net:40534->imta-po.sys.comcast.net:smtp (CLOSE_WAIT)
perl 4600 apache 18u IPv4 22086024 0t0 TCP dedicated.soloservices.net:40536->imta-po.sys.comcast.net:smtp (CLOSE_WAIT)
perl 4600 apache 19u IPv4 22086046 0t0 UDP *:60891
perl 4600 apache 20u IPv4 22085979 0t0 TCP dedicated.soloservices.net:48622->mta-v4.mail.vip.ne1.yahoo.com:smtp (CLOSE_WAIT)
perl 4600 apache 21u sock 0,6 0t0 22084905 can't identify protocol
perl 4600 apache 22u IPv4 22086049 0t0 TCP dedicated.soloservices.net:55754->mtain-a-atc-b.mx.aol.com:smtp (CLOSE_WAIT)
perl 4600 apache 23u sock 0,6 0t0 22084902 can't identify protocol
perl 4600 apache 24u IPv4 22086047 0t0 UDP *:35954
perl 4600 apache 25u IPv4 22086051 0t0 TCP dedicated.soloservices.net:55758->mtain-a-atc-b.mx.aol.com:smtp (CLOSE_WAIT)
perl 4600 apache 26u IPv4 22086044 0t0 UDP *:46542
perl 4600 apache 27u IPv4 22086053 0t0 TCP dedicated.soloservices.net:55762->mtain-a-atc-b.mx.aol.com:smtp (CLOSE_WAIT)
perl 4600 apache 28u IPv4 22086052 0t0 TCP dedicated.soloservices.net:55760->mtain-a-atc-b.mx.aol.com:smtp (CLOSE_WAIT)
perl 4600 apache 29u IPv4 22086054 0t0 TCP dedicated.soloservices.net:55764->mtain-a-atc-b.mx.aol.com:smtp (CLOSE_WAIT)
perl 4600 apache 30u IPv4 22086055 0t0 TCP dedicated.soloservices.net:55766->mtain-a-atc-b.mx.aol.com:smtp (CLOSE_WAIT)
perl 4600 apache 31u sock 0,6 0t0 22085418 can't identify protocol
perl 4600 apache 32u IPv4 22085770 0t0 TCP dedicated.soloservices.net:52386->mta-v1.mail.vip.bf1.yahoo.com:smtp (CLOSE_WAIT)
perl 4600 apache 33u IPv4 22086056 0t0 UDP *:44916
perl 4600 apache 35u IPv4 22085769 0t0 TCP dedicated.soloservices.net:52384->mta-v1.mail.vip.bf1.yahoo.com:smtp (CLOSE_WAIT)
perl 4600 apache 38u sock 0,6 0t0 22085416 can't identify protocol
perl 4600 apache 39u sock 0,6 0t0 22085420 can't identify protocol
perl 4600 apache 40u sock 0,6 0t0 22085434 can't identify protocol
perl 4600 apache 44u sock 0,6 0t0 22085436 can't identify protocol
perl 4600 apache 50u IPv4 22085448 0t0 TCP dedicated.soloservices.net:36992->mx1.hotmail.com:smtp (CLOSE_WAIT)
perl 4600 apache 51r FIFO 0,8 0t0 923850 pipe
perl 4600 apache 52w FIFO 0,8 0t0 923850 pipe
perl 4600 apache 53r FIFO 0,8 0t0 923851 pipe
perl 4600 apache 54w FIFO 0,8 0t0 923851 pipe
perl 4600 apache 55w REG 253,3 4251 1337727 /var/log/httpd/mod_jk.log
perl 4600 apache 56u REG 253,3 1024 1313820 /var/log/httpd/jk-runtime-status.12866 (deleted)
perl 4600 apache 57u REG 253,3 1 1313883 /var/log/httpd/jk-runtime-status.12866.lock (deleted)
perl 4600 apache 63u IPv4 22085832 0t0 TCP dedicated.soloservices.net:48376->mta-v4.mail.vip.ne1.yahoo.com:smtp (CLOSE_WAIT)
perl 4600 apache 64u IPv4 22085888 0t0 TCP dedicated.soloservices.net:48472->mta-v4.mail.vip.ne1.yahoo.com:smtp (CLOSE_WAIT)
perl 4600 apache 108u IPv4 22085227 0t0 UDP *:56279
qmail-rsp 21406 qmailr txt REG 253,3 14600 1468991 /var/qmail/bin/qmail-rspawn
---------- Post updated at 04:32 PM ---------- Previous update was at 04:21 PM ----------
Also checked every instance pf perl in users folders cgi-bin as well as html folders. None found.
Hello,
When I'm trying to track down rogue processes on a Linux system, I find the /proc filesystem valuable. If you have full shell access (and it seems you do), and the rogue Perl process is still running with PID 4600 (or if you can see what its current PID is), try doing ls -l /proc/4600 and ls -l /proc/4600/fd/ . This might reveal something of the directory that the underlying process is stored in, or at least give you some clues.
In that 'top' listing, I also have to say I don't much like the look of PIDs 4599, 4334 and 4552. Basically you should pay close attention to any process that's owned by the user 'apache' but claims to be anything other than 'httpd'.
---------- Post updated at 10:28 PM ---------- Previous update was at 10:25 PM ----------
Hi,
Also, one other quick thought - have a good look through /tmp , /var/tmp and /var/run . Especially look for hidden files (files whose name starts with a dot) by means of ls -a . The '-a' flag shows such files in a directory listing, and can be combined with other flags such as '-l'.
tried to do the
1s -1 /proc/4600
and the other. Terminal said there was no such thing as 1s. I am using -bash: in front of everything but the| grep perl revealed 2 instances of perl.
usr/bin/perl and usr/sbin/hspc-plugin-rpc.fcgi which contained
use strict;
use CGI::Fast();
use Pod::WSDL;
use SOAP::Lite;
use SOAP::Transport::HTTP;
use HSPC::Config;
use HSPC::Plugin::SOAP::Common;
use HSPC::Plugin::SOAP::Struct;
use HSPC::Plugin::SOAP::PP;
$ENV{"plugin_soap_common"} = HSPC::Plugin::SOAP::Common->new();
my $remote_uri = "https://".$CONFIG{"SERVER_NAME"}.":".$CONFIG{"SERVER_PORT"}."/HSPC/Plugin/SOAP";
my $local_uri = "https://127.0.0.1:".$CONFIG{"SERVER_PORT"}."/HSPC/Plugin/SOAP";
my $cgi = SOAP::Transport::HTTP::FCGI
-> dispatch_with({
$remote_uri."/WebServices" => "HSPC::Plugin::SOAP::WebServices",
$local_uri."/WebServices" => "HSPC::Plugin::SOAP::WebServices",
$remote_uri."/Common" => "HSPC::Plugin::SOAP::Common",
$local_uri."/Common" => "HSPC::Plugin::SOAP::Common",
$remote_uri."/PP" => "HSPC::Plugin::SOAP::PP",
$local_uri."/PP" => "HSPC::Plugin::SOAP::PP",
$remote_uri."/PM" => "HSPC::Plugin::SOAP::PM",
$local_uri."/PM" => "HSPC::Plugin::SOAP::PM",
$remote_uri."/DM" => "HSPC::Plugin::SOAP::DM",
$local_uri."/DM" => "HSPC::Plugin::SOAP::DM",
})
-> handle
;
0;
I have no idea if this is suppose to be here or not.
Can I just check - it looks from what you've said that you're trying to type 1s (that's a number one followed by a lower-case letter 'S') rather than ls (that's a lower-case letter 'L' followed by a lower-case letter 'S', which is the correct command).
If that's what you've been doing, could you try again with the correct command name and see what happens please ?
If you have been typing it correctly then your system must be quite badly damaged or missing some very fundamental binaries, since the ls command is pretty much as common as it gets on any UNIX-style system.
You are right. I was doing it wrong. It did look like a 1 to me...so this is what I got:
[root@dedicated ~]# ls -l /proc/4600
ls: cannot access /proc/4600: No such file or directory
[root@dedicated ~]# ls -l/proc/4600/fd
ls: invalid option -- '/'
Try `ls --help' for more information.
[root@dedicated ~]# ls -l /proc/4600/fd
ls: cannot access /proc/4600/fd: No such file or directory
[root@dedicated ~]# ls -a
. echo .odbc.ini
.. findbot.pl parallels
1 ghosttest .pki
anaconda-ks.cfg ghosttest.c plk
.autoinstaller id_rsa.pub psasem.sem
.bash_history install_keys.sh .rnd
.bash_logout install.log run.pl
.bash_profile install.log.syslog .spamassassin
.bashrc .lesshst .ssh
bash-shellshocker mysql-community-release-el6-5.noarch.rpm strace.log
.cshrc .mysql_history .tcshrc
Tech support got back to me last night said it was a brute force attack (?) and that my install of "ban to fail" stopped it. But to me it looks as if it was run from "inside my computer". It slowed the server down to a crawl. also legitimate emails have stopped being able to send to comcast, Verizon, yahoo,etc. Have contacted them but they say we are not on their black list. Just can't figure this out. I have a minimal tech support package so they say "You have this....and you need to do this....good luck....Oh and by the way you can upgrade for $$$".
Hi,
The error messages you got from 'ls' would mean that that PID 4600 no longer exists - in other words, the process with ID 4600 has since exited. These kinds of things only tend to hang around for so long, and you really need to catch them right in the act to have any decent chance of easily tracking them down. If you see any other suspicious processes still (to recap, that would be processes owned by 'apache' but which claim to be anything other than 'httpd') then have a look at their entries in the /proc filesystem in the same way. But most likely it's all over and done with by now.
Unfortunately, the bigger problem you have here is not in fact the rogue scripts themselves, but the question of how they came to be on your system in the first place. In my experience, an attacker finds something they can exploit, like a file upload form without sufficient security protection, or some other script that they can exploit to make it upload things to a globally-writable location on the server. Once they upload their script, they then run it, it hangs about for a bit while it does its thing, then exits.
So the more important thing you have to do here is figure out how the attackers got these scripts on your system. The fact that they might still be somewhere on your server is certainly a problem, but a bigger problem is the fact that they were able to be on your server at all in the first place. In all likelihood you have a security hole somewhere that could be exploited by the same attacker again, or an entirely different attacker in the future.
I must also tell you that one of my my friends newly installed website which was a wordpress site from another host, Web.com, which i let them install...big mistake...had some bad code and I see user petra has php-cgi running where there shouldn't be. So, this may be the culprit. tried running the proc command on that:
764 petra195 30 10 130m 37m 29m S 2.3 2.0 0:01.20 php-cgi
606 petra195 30 10 130m 37m 29m S 0.0 2.0 0:01.08 php-cgi
and got nothing. He should have no cgi scripts running. How do I find and remove them if they are buried in the install?
Hi,
Ah, WordPress. So many times I've seen compromised sites and servers that began with an incorrectly-configured, inadequately-secured or out-of-date WordPress install. Seeing 'php-cgi' by itself isn't necessarily sinister. WordPress is PHP based, and so any page you access will certainly spawn a PHP process somehow.
If your server is running Plesk (and from the 'sw-engine-fpm' process in one of your earlier process lists I'm pretty sure it must be), then this is a fairly standard way for Plesk to deal with PHP requests. Any request for a page that needs PHP to handle it will result in a 'php-cgi' process being spawned by Plesk, if it's configured to handle PHP in its usual way.
But I'd definitely start with that WordPress install, in terms of a suspect. At a bare minimum ensure it's up-to-date, that all its plugins are up-to-date (and that they are all legitimate and not malicious or suspicious plugins), that all your WordPress users and admins are ones you actually expect to exist , that they all have good strong passwords set, and so on and so forth. Pay very close attention to anything that allows file uploads in particular. Having anything that allows uploads of any sort without a username and password being required is a disaster waiting to happen, pretty much.
Well, I thought I took care of that by making him start from scratch. I dumped the entire site, including the database, did a clean install of wordpress myself from wordpress.org and only installed plugins for the official site via their built in tools. No ftp of plugins from strange places. So, I thought he was clean.
You can only upload into the media section from the admin section of the site so I think he should be fine there, unless he has malicious code attached to a graphic? Is that possible?
Hi,
It certainly sounds like you've been sensible in how you've set up WordPress, from what you describe. And whilst it would be possible to conceal a malicious script in an image, this isn't commonly seen, and tends to be a result of far more subtle and advanced compromises than what you appear to be experiencing.
The main thing I suggest right now is to see if there are still any suspicious processes on your system at the moment. If there are, then you need to find out as much as you can about them before they exit or are killed off. If there aren't, then your next best avenue of investigation would be to look at your Web server logs for anything that appears out-of-the-ordinary.
On a Plesk-style system (if that is indeed what you have), these are typically stored as /var/www/vhosts/domain.com/logs/access_log , where domain.com should of course be replaced with your own Web site's domain. If it's a WordPress site, then any mention of the Perl interpreter or Perl scripts in those logs would constitute an abnormal finding.
I do have Plesk and it is very user friendly. I have installed "watchdog" and it runs a script periodically and sends me an email. No viruses found but it did come up with this:
[14:28:03] Info: SCAN_MODE_DEV set to 'THOROUGH'
[14:28:04] Checking /dev for suspicious file types [ Warning ]
[14:28:04] Warning: Suspicious file types found in /dev:
[14:28:04] /dev/shm/R1Soft-SHM-ZocPNFWuCcEl2rrN: data
[14:28:04] Checking for hidden files and directories [ Warning ]
[14:28:04] Warning: Hidden directory found: /dev/.udev
[14:28:04] Warning: Hidden file found: /usr/share/man/man1/..1.gz: gzip compressed data, from Unix, max compression
[14:28:04] Warning: Hidden file found: /usr/share/man/man5/.k5login.5.gz: gzip compressed data, from Unix, max compression
[14:28:04] Warning: Hidden file found: /usr/share/man/man5/.k5identity.5.gz: gzip compressed data, from Unix, max compression
[14:28:04] Warning: Hidden file found: /usr/bin/.fipscheck.hmac: ASCII text
[14:28:04] Warning: Hidden file found: /usr/bin/.ssh.hmac: ASCII text
[14:28:04] Warning: Hidden file found: /usr/sbin/.sshd.hmac: ASCII text
Hello,
That's certainly a good idea in general, to run something like 'watchdog' or 'tripwire'. I don't think anything it's found is particularly worrisome, and is all stuff I'd more or less expect to be there, depending on what you have installed.
The main thing I'd definitely do is look at what processes are running at the moment, and investigate the /proc entries for any suspicious ones that are still there. Next, read through your Web logs for anything that looks abnormal around the time that the incident is alleged to have began. With any luck there's a clear trace in the Web logs that will show you where the dodgy scripts were stored, and perhaps even how they came to be uploaded.
Looked at the access logs for "petra" found this during the attack:
66.249.65.5 - - [23/Feb/2017:22:36:26 -0500] "GET /1714/9292rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 301 395 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.73.63 - - [23/Feb/2017:22:36:27 -0500] "GET /1714/9292rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 404 10615 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.73.63 - - [23/Feb/2017:22:36:35 -0500] "GET /1715/1478rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 404 10615 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.5 - - [23/Feb/2017:22:36:40 -0500] "GET /1717/4280rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 301 395 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.73.63 - - [23/Feb/2017:22:36:41 -0500] "GET /1717/4280rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 404 10615 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.5 - - [23/Feb/2017:22:36:54 -0500] "GET /1714/1680rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 301 395 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.73.63 - - [23/Feb/2017:22:36:55 -0500] "GET /1714/1680rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 404 10615 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.9 - - [23/Feb/2017:22:37:08 -0500] "GET /1702/8357rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 301 395 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.73.2 - - [23/Feb/2017:22:37:09 -0500] "GET /1702/8357rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 404 10615 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.7 - - [23/Feb/2017:22:37:22 -0500] "GET /1714/8647rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 301 395 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.73.63 - - [23/Feb/2017:22:37:23 -0500] "GET /1714/8647rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 404 10615 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.5 - - [23/Feb/2017:22:37:36 -0500] "GET /1707/3950rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 301 395 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.73.63 - - [23/Feb/2017:22:37:37 -0500] "GET /1707/3950rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 404 10615 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.5 - - [23/Feb/2017:22:37:50 -0500] "GET /1704/4965rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 301 395 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.73.63 - - [23/Feb/2017:22:37:51 -0500] "GET /1704/4965rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 404 10615 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.9 - - [23/Feb/2017:22:38:04 -0500] "GET /1704/6309rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 301 395 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.73.2 - - [23/Feb/2017:22:38:05 -0500] "GET /1704/6309rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 404 10615 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.5 - - [23/Feb/2017:22:38:18 -0500] "GET /1704/4054rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 301 395 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.73.63 - - [23/Feb/2017:22:38:19 -0500] "GET /1704/4054rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 404 10615 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.5 - - [23/Feb/2017:22:38:32 -0500] "GET /1704/4079rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 301 395 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.73.63 - - [23/Feb/2017:22:38:33 -0500] "GET /1704/4079rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 404 10615 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.5 - - [23/Feb/2017:22:38:46 -0500] "GET /1702/4865rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 301 395 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.73.63 - - [23/Feb/2017:22:38:47 -0500] "GET /1702/4865rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 404 10615 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
72.4.160.86 - - [23/Feb/2017:22:38:59 -0500] "GET / HTTP/1.0" 301 255 "-" "-"
66.249.65.5 - - [23/Feb/2017:22:39:00 -0500] "GET /1707/6902rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 301 395 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.73.63 - - [23/Feb/2017:22:39:01 -0500] "GET /1707/6902rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 404 10615 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.9 - - [23/Feb/2017:22:39:14 -0500] "GET /1704/8694rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 301 395 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.73.63 - - [23/Feb/2017:22:39:15 -0500] "GET /1704/8694rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 404 10615 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.5 - - [23/Feb/2017:22:39:28 -0500] "GET /1708/9940rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 301 395 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.73.63 - - [23/Feb/2017:22:39:29 -0500] "GET /1708/9940rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 404 10615 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.5 - - [23/Feb/2017:22:39:42 -0500] "GET /1709/4377rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 301 395 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.73.63 - - [23/Feb/2017:22:39:43 -0500] "GET /1709/4377rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 404 10615 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.5 - - [23/Feb/2017:22:39:56 -0500] "GET /1711/7066rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 301 395 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.73.63 - - [23/Feb/2017:22:39:57 -0500] "GET /1711/7066rgpmuw/-guwuiwp-jmf-snj-756278-rgs/wwq HTTP/1.0" 404 10615 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.65.9 - - [23/Feb/2017:22:40:05 -0500] "GET /an
Hi,
I don't think that what you've found is likely to be terribly relevant. Unless the User-Agent has been faked, that's just normal Google crawling.
You're looking for anything that appears to concern a Perl script. POST operations are also probably of particular interest, rather than GETs. At least one POST was probably involved at one point to invoke the malicious script.
Note that there will be separate log files for each site on your server, so you will need to check each Apache access_log that exists on your server in case the exploit was started from another site or location than your WordPress site.
Doing that now but since that was the wordpress install I thought I'd look at that one first. Lots for logs to go through.
---------- Post updated at 09:23 AM ---------- Previous update was at 09:15 AM ----------
This one is kinda different:
195.133.48.121 - - [23/Feb/2017:00:37:19 -0500] "HEAD / HTTP/1.0" 200 300 "http://bettingtips.pro/" "Mozilla/4.0 (compatible; MSIE3.00; Windows 2006)"
195.133.48.150 - - [23/Feb/2017:00:37:20 -0500] "HEAD / HTTP/1.0" 200 300 "http://www.mai-club.ru/otzyivyi.html" "Mozilla/4.0 (compatible; MSIE3.00; Windows 2009)"
---------- Post updated at 09:33 AM ---------- Previous update was at 09:23 AM ----------
and:
187.190.245.207 - - [23/Feb/2017:00:45:49 -0500] "POST /xmlrpc.php HTTP/1.0" 404 10636 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
187.190.245.207 - - [23/Feb/2017:00:45:50 -0500] "POST /wp-login.php HTTP/1.0" 200 3937 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
195.133.147.89 - - [23/Feb/2017:00:56:28 -0500] "HEAD / HTTP/1.0" 200 300 "http://worldcuisine.pro/" "Mozilla/2.0 (compatible; MSIE7.00; Windows 2006)"
193.124.131.168 - - [23/Feb/2017:00:56:28 -0500] "HEAD / HTTP/1.0" 200 300 "http://banket-restoran.ru/osobennosti-svadby-na-kipre.html" "Mozilla/5.0 (compatible; MSIE3.00; Windows 2002)"
103.21.44.26 - - [23/Feb/2017:01:09:43 -0500] "POST /xmlrpc.php HTTP/1.0" 404 10636 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
103.21.44.26 - - [23/Feb/2017:01:09:44 -0500] "POST /wp-login.php HTTP/1.0" 200 3937 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
195.133.144.24 - - [23/Feb/2017:01:30:45 -0500] "HEAD / HTTP/1.0" 200 300 "http://nonbox.ru/metod-borby-s-alkogolnoj.html" "Mozilla/2.0 (compatible; MSIE4.00; Windows 2009)"
195.133.147.6 - - [23/Feb/2017:01:31:58 -0500] "HEAD / HTTP/1.0" 200 300 "https://nochuem.ru" "Mozilla/3.0 (compatible; MSIE2.00; Windows 2003)"
So you get the idea. These are all from Petra.
Hi,
Aside from perhaps the POST /xmlrpc.php entries, nothing there is jumping out at me as being majorly sinister (so long as the IP address from which the POST /wp-login.php originated was expected, naturally). There have been some longstanding issues with WordPress security around xmlrpc.php, in certain configurations.
If you'll never actually need the functionality it offers (which more or less revolves around letting you update your blog remotely via certain kinds of third-party and mobile apps), then you can probably safety disable access to that file.
All in all though, I think you've probably still to find your "smoking gun" here, to so speak.
Also, sorry to ask the same thing again, but it is rather important: are you sure there are currently no sinister or unexpected processes running on your server right now ?
Nothing right now.
---------- Post updated at 11:07 AM ---------- Previous update was at 10:56 AM ----------
This is what my host admin said currently:
Thank you for getting back to us.
The 'HEAD / HTTP/1.0" 200 300' requests you are seeing are multi-choice requests, which require more clarification what sources was actually request. These are typically generated by bad redirects from outside sources via incorrect links.
Regarding the 'POST /wp-login.php HTTP/1.0" 200' and 'POST /xmlrpc.php HTTP/1.0" 404' logs, those are most likely due to someone trying to brute-force your WordPress website. This is an indication of an XML-RPC attack. I would highly advise you to either password protect wp-login.php or wp-admin, in order to prevent outside access to these crucial parts of your website.
Depending on how well you have secured your WordPress site it may eventually lead to the website being compromised. As I see you are using Fail2Ban for your wordpress sites, it should not be an issue.
So, it should not be an issue? have I blown this all out of proportion and worried for nothing, not to mention everybody's time here. If so, I apologize.
But I'm glad to see this forum is active and willing to help.