OK, so I've been learning my way through Fedora trying to progress to LFS and FreeBSD. I have a Fedora 14 machine running Apache 2.2.17, and about 2 days ago, I came across the server and saw a black screen blazing through text so fast I couldn't read it. I didn't know if it was a crash or I'd been compromised, so I pulled the ethernet cable and hit the reset button. Upon restarting I was greeted with:
Inodes that were part of a corrupted orphan linked list found
/dev/mapper/vg_192-lv_root: UNEXPECTED INCONSISTENCY; RUN fsck manually (i.e., without -a or -p options)
I ran fsck and selected y for about 15 to 30 error fixes. I would have written them down but I'd been planning on rebuilding this box. Now, however, I'd like to know the cause of this problem, so I've been googling access log messages, syslog messages, etc. Just now I stuck in some external firewall (router) rules that only allow traffic between the apache box and 3 IP addresses that I use (home and work) I found the following (edited) in the firewall log:
Oct 24 19:58:07 2011 TCP 192.168.1.xx:60664->204.141.87.16:80 on ixp0 [repeated 6 times, last time on Oct 24 19:59:40 2011]
Oct 24 19:54:57 2011 TCP 192.168.1.xx:47455->204.141.87.11:80 on ixp0 [repeated 6 times, last time on Oct 24 19:56:31 2011]
Oct 24 19:54:06 2011 TCP 192.168.1.xx:47454->204.141.87.11:80 on ixp0 [repeated 5 times, last time on Oct 24 19:54:52 2011]
Oct 24 19:50:57 2011 TCP 192.168.1.xx:60661->204.141.87.16:80 on ixp0 [repeated 6 times, last time on Oct 24 19:52:30 2011]
Oct 24 19:48:57 2011 TCP 192.168.1.xx:60660->204.141.87.16:80 on ixp0 [repeated 6 times, last time on Oct 24 19:50:30 2011]
Oct 24 19:47:57 2011 TCP 192.168.1.xx:60659->204.141.87.16:80 on ixp0 [repeated 5 times, last time on Oct 24 19:48:42 2011]
Oct 24 19:47:27 2011 TCP 192.168.1.xx:60658->204.141.87.16:80 on ixp0 [repeated 4 times, last time on Oct 24 19:47:48 2011]
Oct 24 19:45:47 2011 TCP 192.168.1.xx:60657->204.141.87.16:80 on ixp0 [repeated 6 times, last time on Oct 24 19:47:20 2011]
they were blocked and logged. It's a US company...don't know why my server is attempting to contact these IPs?
some other info. Server had been up and running with phpbb 3.0.9 (no registered users other than myself) for about 4 months. I looked over the inactive users list and filtered all of the IP blocks of those users (Russia and Ukraine) in the firewall about 2 nights before this happened. Running vsftp 2.3.4 with 2 users, one user's home directory root of one of the virual hosts, and the other was /home/user. Both could log in locally and move up outside their home directories. I was in the middle of figuring out how to lock down the one user (vhost home dir) and was going to remove FTP access for the other but I forgot to.
I know I was littered with security holes, and plan on addressing them before the new one goes online. I still have this one running as described, but don't plan on unleashing it. I would like to find out if it has been broken into or not before I start over. Anyone have any idea where to check, or why it's trying to connect to those 2 ip addresses?
thank you for your time and for sharing your priceless knowledge
---------- Post updated at 09:33 PM ---------- Previous update was at 09:26 PM ----------
Some more firewall security log:
Oct 24 21:28:55 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:52906->128.63.2.53:53 on ixp0
Oct 24 21:28:56 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:50508->192.228.79.201:53 on ixp0
Oct 24 21:28:56 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:29322->192.203.230.10:53 on ixp0
Oct 24 21:28:56 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:44569->192.203.230.10:53 on ixp0
Oct 24 21:28:56 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:5220->192.228.79.201:53 on ixp0
Oct 24 21:28:56 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:14340->128.8.10.90:53 on ixp0
Oct 24 21:28:56 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:44225->192.112.36.4:53 on ixp0
Oct 24 21:28:56 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:63708->192.36.148.17:53 on ixp0
Oct 24 21:28:57 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:37912->192.203.230.10:53 on ixp0
Oct 24 21:28:57 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:25084->192.112.36.4:53 on ixp0
Oct 24 21:28:57 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:8359->192.112.36.4:53 on ixp0
Oct 24 21:28:57 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:32749->192.112.36.4:53 on ixp0
Oct 24 21:28:57 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:5118->192.203.230.10:53 on ixp0
Oct 24 21:28:57 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:39603->199.7.83.42:53 on ixp0
Oct 24 21:28:58 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:18159->202.12.27.33:53 on ixp0
Oct 24 21:28:58 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:14484->128.8.10.90:53 on ixp0
Oct 24 21:28:58 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:2248->193.0.14.129:53 on ixp0
Oct 24 21:28:58 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:64654->193.0.14.129:53 on ixp0
Oct 24 21:28:58 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:51058->193.0.14.129:53 on ixp0
Oct 24 21:28:58 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:8535->193.0.14.129:53 on ixp0
Oct 24 21:28:58 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:60558->193.0.14.129:53 on ixp0
Oct 24 21:28:58 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:38959->193.0.14.129:53 on ixp0
Oct 24 21:28:59 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:28654->192.36.148.17:53 on ixp0
Oct 24 21:28:59 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:54992->192.5.5.241:53 on ixp0
Oct 24 21:28:59 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:62956->198.41.0.4:53 on ixp0
Oct 24 21:28:59 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:41303->128.63.2.53:53 on ixp0
Oct 24 21:28:59 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:31276->192.5.5.241:53 on ixp0
Oct 24 21:28:59 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:14432->199.7.83.42:53 on ixp0
Oct 24 21:28:59 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:3762->192.5.5.241:53 on ixp0
Oct 24 21:29:00 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:63866->128.63.2.53:53 on ixp0
Oct 24 21:29:00 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:1180->192.33.4.12:53 on ixp0
Oct 24 21:29:01 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:25832->128.63.2.53:53 on ixp0
Oct 24 21:29:01 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:62628->199.7.83.42:53 on ixp0
Oct 24 21:29:01 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:34998->192.36.148.17:53 on ixp0
Oct 24 21:29:01 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:53695->198.41.0.4:53 on ixp0
Oct 24 21:29:01 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:49673->192.58.128.30:53 on ixp0
Oct 24 21:29:01 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:8749->202.12.27.33:53 on ixp0
Oct 24 21:29:01 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:28551->128.8.10.90:53 on ixp0
Oct 24 21:29:02 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:21114->192.203.230.10:53 on ixp0
Oct 24 21:29:02 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:29879->193.0.14.129:53 on ixp0
Oct 24 21:29:02 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:4627->192.5.5.241:53 on ixp0
Oct 24 21:29:02 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:31509->192.58.128.30:53 on ixp0
Oct 24 21:29:03 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:49220->202.12.27.33:53 on ixp0
Oct 24 21:29:03 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:3657->192.33.4.12:53 on ixp0
Oct 24 21:29:03 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:44276->192.58.128.30:53 on ixp0
Oct 24 21:29:03 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:14984->192.5.5.241:53 on ixp0
Oct 24 21:29:03 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:16620->192.228.79.201:53 on ixp0
Oct 24 21:29:04 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:43240->192.58.128.30:53 on ixp0
Oct 24 21:29:04 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:48719->202.12.27.33:53 on ixp0
Oct 24 21:29:04 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:52690->198.41.0.4:53 on ixp0
Oct 24 21:29:04 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:7414->128.8.10.90:53 on ixp0
Oct 24 21:29:04 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:16525->192.228.79.201:53 on ixp0
Oct 24 21:29:04 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:4261->199.7.83.42:53 on ixp0
Oct 24 21:29:05 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:27184->192.203.230.10:53 on ixp0
Oct 24 21:29:05 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:44511->192.112.36.4:53 on ixp0
Oct 24 21:29:06 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:4327->199.7.83.42:53 on ixp0
Oct 24 21:29:08 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:16148->193.0.14.129:53 on ixp0
Oct 24 21:29:09 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:40769->128.63.2.53:53 on ixp0
Oct 24 21:29:11 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:38067->202.12.27.33:53 on ixp0
Oct 24 21:29:12 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:64440->192.5.5.241:53 on ixp0
Oct 24 21:29:14 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:52151->192.58.128.30:53 on ixp0
Oct 24 21:29:15 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:6116->198.41.0.4:53 on ixp0
Oct 24 21:29:17 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:9749->193.0.14.129:53 on ixp0
Oct 24 21:29:19 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:12706->192.58.128.30:53 on ixp0
Oct 24 21:29:21 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:65001->192.5.5.241:53 on ixp0
Oct 24 21:29:23 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:30482->128.63.2.53:53 on ixp0
Oct 24 21:29:25 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:40425->202.12.27.33:53 on ixp0
Oct 24 21:29:27 2011 Outbound Traffic Blocked - Advanced Filter Rule UDP 192.168.1.xx:14780->198.41.0.4:53 on ixp0
Oct 24 21:29:58 2011 Outbound Traffic Blocked - Advanced Filter Rule TCP 192.168.1.xx:47461->204.141.87.11:80 on ixp0