Filter records from a log file based on timestamp

UNIX for Beginners Q & A
1

Dear Experts,

I have a log file that contains a timestamp, I would like to filter record from that file based on timestamp. For example refer below file -
cat sample.txt

Jan 19 20:51:48 mukul-Vostro-14-3468 systemd: pam_unix(systemd-user:session): session opened for user root by (uid=0)
Jan 19 20:54:48 mukul-Vostro-14-3468 su[806]: pam_unix(su:session): session closed for user root
Jan 19 20:57:48 mukul-Vostro-14-3468 systemd-logind[1048]: Removed session c49.
Jan 19 20:59:48 mukul-Vostro-14-3468 su[854]: Successful su for root by root
Jan 19 20:57:48 mukul-Vostro-14-3468 su[854]: + ??? root:root
Jan 19 21:05:48 mukul-Vostro-14-3468 su[854]: pam_unix(su:session): session opened for user root by (uid=0)
Jan 19 21:10:48 mukul-Vostro-14-3468 systemd-logind[1048]: New session c50 of user root.
Jan 19 21:12:48 mukul-Vostro-14-3468 systemd: pam_unix(systemd-user:session): session opened for user root by (uid=0)
Jan 19 21:15:49 mukul-Vostro-14-3468 su[854]: pam_unix(su:session): session closed for user root

Out of this file i would like to filter record before 21:10 hrs.

Any help is appreciated.

2

Any attempts / ideas / thoughts from your side? Did you consider searching tis site, or looking into the links at the bottom of this thread?

Are the entries sorted by timestamp?

3

Hello Rudic,

Thanks for your quick revert. Yes the entries are sorted. Haven't gave it a shot since i wasn't sure where to start with. One thing that i got to know is that the timestamp need to be converted into string first before i apply any sort of comparison.

4

OK; give it a shot and report back the errors and/or shortfalls you encounter.

1 Like
5

Dear Experts,
This is what I've tried by converting into epoch time and it's giving me the desired result however it seems to me like this is not the standard way of doing this, do you mind taking a look and improve this solution pls.

dt1=`date -d 'Jan 19 21:15:00' +%s`
#echo $dt1
while read line
do
	tmstmp=$(echo $line | cut -c '1-15' )
	#echo $tmstmp
	dt2=`date -d "$tmstmp" +%s`
	#echo  $dt2
	echo "Ref time in epoch is $dt1"
	echo "Timestamp from file is $tmstmp"
	echo "Timestamp from file in epoch is $dt2"
	if [[ $dt2 -le $dt1 ]]
	then
		echo "This event occured before 21:15, inserting into new log file"
		echo $line >> filter.txt
	else
		continue 
		
	fi		
	
done < auth.log
6

That looks like a good approach.

You could get some more efficiency by not calling the date external on each input line and using bash to convert the date time to an mmddHHMMSS number for direct comparison like this:

#!/bin/bash
function convdtime {
    w="JanFebMarAprMayJunJulAugSepOctNovDec"
    t=${w%%$1*}
    printf -v when "%d%02d%02d%02d%02d\n"  $((${#t}/3 + 1)) $2 ${3//:/ }
}

convdtime Jan 19 21:15:00
start=$when

while read mth day time rest
do
   convdtime $mth $day $time
   if [[ $when -le $start ]]
   then
       echo "This event occured before 21:15, inserting into new log file"
       echo "$mth $day $time $rest" >> filter.txt
   else
      break
   fi
done < auth.log

If file is sorted you can save processing additional lines by using break instead of continue once you are past the target datetime.

Edit: This will have issues around end of calendar year where Dec 31 will appear to be greater than Jan 1

1 Like
7

For the fun of it; the (admitted lengthy) "command substitution" finds the last line prior to the date/time given in DT1 on which sed needs to quit:

DT1="Jan 19 21:10:00"
sed "/$(echo "$DT1" | cut -d" " -f1-3 - file1 | date -f- +"%F %T" | { read T1; { cat; echo $T1; } | sort | grep -B1 "$T1" | head -1 | date -f- +"%b %d %H:%M:%S"; } )/q" file
Jan 19 20:51:48 mukul-Vostro-14-3468 systemd: pam_unix(systemd-user:session): session opened for user root by (uid=0)
Jan 19 20:54:48 mukul-Vostro-14-3468 su[806]: pam_unix(su:session): session closed for user root
Jan 19 20:57:48 mukul-Vostro-14-3468 systemd-logind[1048]: Removed session c49.
Jan 19 20:59:48 mukul-Vostro-14-3468 su[854]: Successful su for root by root
Jan 19 20:57:48 mukul-Vostro-14-3468 su[854]: + ??? root:root
Jan 19 21:05:48 mukul-Vostro-14-3468 su[854]: pam_unix(su:session): session opened for user root by (uid=0)
1 Like