Even if it does not directly help in this case, I'd like to mention Linux Audit, because it's really interesting:
One can audit all commands entered by anyone with these 2 rules:
auditctl -a always,exit -F arch=b32 -S execve -F key=audit_all_commands
auditctl -a always,exit -F arch=b64 -S execve -F key=audit_all_commands
To make it persistent, you need to place the commands in the auditd-rule files.(For Red Hat they are located in /etc/audit/rules.d)
...and then use ausearch
for investigation of /var/log/audit/audit.log
, or just grep it.
Interesting thing about Linux-Audit:
If you do su
or sudo
Linux-Audit tracks your original user anyway.
And for this investigation it won't help you: Because you need to have the rules in place before the event to be investigated happens.
Here's an example of the two most interesting lines of what you might have in the log(/var/log/audit/audit.log):
type=SYSCALL msg=audit(1575314015.062:20): arch=c000003e syscall=59 success=yes exit=0 a0=55c7fbbc4e20 a1=55c7fbbc43a0 a2=55c7fbbc86a0 a3=2 items=2 ppid=1 pid=832 auid=1004 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="reboot" exe="reboot" key="audit_all_commands"
type=EXECVE msg=audit(1575314015.062:20): argc=1 a0="reboot"
Auditd is very noisy in logging. That means one possibly has to dig a lot log records.
As you see above the key audit_all_commands is in the line. So you can grep for your configured string. And uid in the first line shows 0 which is root. But the auid-value shows 1004, which is the real user id(which originally logged on to the system before using su or sudo. So it's a good idea to restrict direct root access and only allow sudo for becoming root).
The second line is linked with the first line through the audit-id field(audit(1575314015.062:20)) and shows you the linked command with parameters executed.
What can you do with Linux-Audit?
- Trace commands with rules to include or exclude specific events
- Trace File System Activities (Access with type: read, write, permission change, execute) on selected files or directory trees
- Trace Standard Linux Management activities(user/group management, Firewall-Configuration, Audit-Log-Config-Changes,...)
- setup logging to remote sites or syslogs
- use it to log important information into it yourself
I worked the last 3 month with it and did a lot with it. For example i created a little perl script which uses inotify & git & audit to log diffs of config file changes into audit log which could then be tracked down to the causing user accounts.
Here's the perl script, if someone cares: [Perl] #!/usr/bin/perl # # - Track files in /etc # - Create audit log - Pastebin.com