Splitting files using awk and reading filename value from input data

I have a process that requires me to read data from huge log files and find the most recent entry on a per-user basis. The number of users may fluctuate wildly month to month, so I can't code for it with names or a set number of variables to capture the data, and the files are large so I don't want to read the it several times.

The entries of interest have a particular string so I can extract just them from the overall log file and I have a way to split the output into separate files on a per-user basis, my plan being to then just read the last line of each files created with tail -1 and the filename giving me the user account in question.

My boss, however, worries about false-positive data matches for my expression (by chance or maliciously) that might try to overwrite a critical file.

My data has a syslog-type date in it which means doing a sort -u is proving tricky too. I've got this far with splitting the data out to files under /tmp/logs as splitlog.rbatte1 or similar but if field 11 were ever */../../etc/passwd then potentially I would be in trouble.

The date is the first three fields and 'as far as I am aware' a valid user name would be in field 11, but ........

A simplified part of the code would be:-

grep "Active transaction started" /var/log/qapplog | awk "{print \$1, \$2, \$3, \$11> \"/tmp/logs/splitlog.\"\$11}"
for userfile in /tmp/logs/splitlog.*
do
   lastrecord=$(tail -1 $userfile)
   printf "User %s last record is %s\n" "$userfile" "$lastrecord"
   .... whatever else here ....
done

I have considered adding tr -d "\/" to strip out the characters, but now that it's been raised, I'm concerned that there may be other things I'm not considering.

Is there a better way to work here, potentially with awk getting the equivalent of basename "$11" or variable substitution in the shell of "${{11}##*/}" ?

Any suggestions welcome. Perhaps there is a better design overall that will find the last entry on a per-user basis. The log is thankfully written in time order, so the last in the file by user name is the last by time already.

Kind regards,
Robin

How about posting a few lines from the input file? Some with, and some without the string of interest.

Not sure if I understood the requirement correctly, but, taking my syslog for a sample, wouldn't this fulfill your needs (redirect printout to a "user" file in /tmp/log if need be):

awk '{split ($5, U, /[[]/); T[U[1]] = $1 " " $2 " " $3} END {for (t in T) print t, T[t]}' /var/log/syslog
systemd-tmpfiles Jun 23 08:47:54
anacron Jun 23 08:46:57
dhclient: Jun 23 18:28:50
dbus Jun 23 12:03:34
CRON Jun 23 21:17:01
kernel: Jun 23 21:18:13
mtp-probe: Jun 23 11:40:12

I think you've got it just right, but it will take me a bit to decipher your code!

Many thanks,
Robin

Just add your Active transaction started as an awk "pattern", and it should fly, shouldn't it? You can even refine the pattern to fulfill your boss' proclivities.

Hello, rbatte1.

Without having an actual input or output representative example from you, I do think RudiC has shown you a valid indication of what it is possible. Definitively, you do not have the need to grep for it.
If is of any help to decipher what it has been shown, allow me to break it in pieces.
Essentially, there are two pieces to process each line, based on the format structure of /var/log/syslog

split ($5, U, /[[]/); # it divides field 5 into smaller parts ([ is the separator) and creates an array to capture the pieces using a variable named U.
T[U[1]] = $1 " " $2 " " $3 # builds an index using the variable T where the index is the first element of U, the split done before.

Once all lines has been processed, display the result:

END {for (t in T) print t, T[t]} # iterate over the index and display what's recorded.

Since, this is done for each line, you'll get multiple hits.
If you want to narrow it down to a particular criteria, let's say: "Active transaction started"

awk '/Active transaction started/ {split ($5, U, /[[]/); T[U[1]] = $1 " " $2 " " $3} END {for (t in T) print t, T[t]}' /var/log/syslog

Limiting the index to only those lines that contain the match.

If you were to post a representative example, I am quite sure we could provide other alternatives, as well.
Also, you might be interested in knowing about an open source project named: ELK
Which it does an amazing job at dealing with all sort of information on logs.

A compromise perhaps bolting several things together.

From the suggestions given and further digging, I've ended up with:-

awk "/Active transaction started/ && /${userregex}/ {print \$1, \$2, \$3 \$11 \
> \"/tmp/logs/splitlog.\"gensub(/[^[:alnum:]._-]/, \"\", \"g\", \$11)}" /var/log/qapplog

for userfile in /tmp/logs/splitlog.*
do
   lastrecord=$(tail -1 $userfile)
   printf "User %s last record is %s\n" "$userfile" "$lastrecord"
   .... whatever else here ....
done

To explain it a bit more:-

• The ${userregex} is a regular expression for all the users we are interested in, so we can exclude testing messages which sadly get written to the same log.
  
  
• The text output is just for checking we've got it, there would be further processing (and that's all fine) This is finding a way to get the last logged entry for each user, so the printf line is just for debug.
  
  
• We removed the grep with the two expressions (both must be satisfied) see this thread
  
  
• The output file(s) are redirected to /tmp/logs/splitlog. appended by the output from the gensub on field 11. The gensub removes all characters from field 11 that are not matched by being alphanumeric, full-stop (period for American English), underscore or hyphen, those all being acceptable characters to build a filename from and sensibly allowed in user account names. We could possibly have apostrophes too, but these have been excluded.

It seems to work for me in testing, but I'd appreciate another few sets of eyes to validate I'm not doing something daft and leaving a gaping hole somewhere.

Robin

You won't get a user name from your for loop but a full path to a splitlog.usernamemod file, and you run multiple processes to get at the last entries.

I'm not sure I understand why you abstain from packing as much processing as possible into the awk script, piping its output (= last entry per user) into a while loop.

And, maybe the "further processing" could - to a large? extent - be included into the awk as well?

That give me food for thought. I'm not the best at awk by a long way, so I will see what I can do.

Thanks,
Robin