Our system was hacked

Someone made a mistake, and left our router wide open, pointing all ports to a SCO 6.0.0 system.
Within 24 hours, the following happened.
The contents of all the files (except tar files) in three directories, one directory on each of three different file systems, were replaced with nulls. None of the inode data was changed, meaning that the output of 'ls -l' was the same before and after. In two of the directories the file permissions were 0664, and in the last, the permissions were 0644 and files owned by root.
I have not been able to find anything in any of the log files to indicate who or when this happened.
Since we had adequate backups there was no long term damage.
Any thoughts would be appreciated.

Turn on logging including router logging and closely monitor the logs.

Create a honeypot to lure the hackers into your logging traps, if you are interested to discover who is responsible.

Good job on keeping your backups up to date!

Look in root's homedir for .history .bash_history or similar files.
Run the history command in the respective shell(s).

Ordinary system logins are listed with the last command.

Consult the system logs, for system access and unusual events.
Is there a su or sudo log in /var/log/ or /var/adm/?
Do you happen to have system accounting (sa) running?

Run netstat -a and look for LISTEN; what services are running that use the ports?
Do these services have extra logs?

How good is your root pw? The longer the better.
Did you switch from the 13byte Unix crypt to another crypt that allows longer pws?

Are you sure your system was hacked at all?
Maybe there was a fatal human error like

tar cf - dir | tar xf -

where the read files are already opened for writing, and such data corruptions can occur.

Here is a high level overview:

When you think you are hacked, you should make sure that all your logfiles are preserved.

It's often best to image the disk and make a full copy of the "hacked" disk before you recover from backup.

Then, you can do your analysis on a mounted "hacked" disk images, since you will lose that information (more than likely) when you recover from backup.

However, if you think you have been hacked; you need to also focus on detection. The triad to think about is:

• Detection
• Recovery
• Prevention

If you just casually recover from backup, you will likely lose a lot of valuable forensic information, so as mentioned, you should make a copy of the hacked disk before you recover from backup.

Then, you should move toward hardening the system to insure your log files are logging on the "maybe hacked" computer and any network devices in the path.

You should not start making changes that would lock out the "hacker" (if there actually was one) without considering if you want to try to trap and trace them. If you block them quickly, the "hacker" will know you are on to them and stop, generally speaking. It's a game of cat and mouse.

So, it's good to set up "honeypots" or any simple way to fool an intruder to thinking they have access, while you log their activities.

A simple example is to write a wrapper around various system commands which logs before it executes. I recently did this with curl on a system I manage. Hackers often use commands like curl or even sh or bash (all os dependent of course) to download and install malicious code, etc.

Or maybe you are not interested in (future) detection?

Then of course you can work on logfile and file system analysis (so you can do a better job at prevention).

One of the cool things I do these days is to use github to backup critical logs and configuration files as well as critical filesystems.

Then, if there is some suspicious activity, I use the private github files to check for changes (file system integrity).

A lot of people do not realize how useful git and github can be for file system analysis, related to "hack analysis". git is not only useful for tracking code changes when developing code. git can be a great filesystem integrity tool in the toolkit of IT security.

Recently, I used github to check the entire filesystem after a suspicious incident and could easily see what file system changes had occurred (in this case, it was none, as I caught the attempted malware injection before it could write to the filesystem ...)

So, whatever you use, you need some way to manage and check the integrity of your filesystem(s) on critical systems.

I don't want to distract you with too much IT security mumbo-jumbo, so let me sign-off before I start another chapter on basic IT security in this post.

Cheers.