Search for text between two time frame using sed

Shell Programming and Scripting
1

I have log files with time stamps. I want to search for text between two time stamp using sed even if the first tme stamp or the last time stamp are not present. For e.g. if i search between 9:30 and 9:40 then it should return text even if 9:30 or 9:40 is not there but between 9:30 and 9:40 is present.
I am using a sed one liner:

sed -n '/7:30:/,/7:35:/p' xyz.log

But it only returns data if both the time stamps are present, it will print everything if one of the time stamp are missing. An if the time is in 12 hr format it will pull data for both AM and PM.
Additionally , i have different time stamps formats for different log files so i need a generic command.
following are some time format examples e.g.

<Jan 27, 2013 12:57:16 AM MST>Jan 29, 2013 8:58:12 AM 2013-01-31 06:44:04,883

One of them contains AM/PM i.e. 12 hr format and other cntains 24 hr format so i have to account for that as well.
I have tried this as weel but it doesnt work:

sed -n -e '/^2012-07-19 18:22:48/,/2012-07-23 22:39:52/p' history.log

Please help

2

Can you post few sample formats including text that you want to extract and desired output?

3

Some of the time formats

--> <Jan 27, 2013 12:57:16 AM MST>
--> Jan 29, 2013 8:58:12 AM
--> 2013-01-31 06:44:04,883

Entry can be anything like:

 
--> 2013-01-31 06:44:04,883 Error in the log file etc...
--> <Jan 27, 2013 12:57:16 AM MST> Logger initialized etc...

Now i want to get everything between 7:30 and 7:40

 
--> 2013-01-28 07:33:57,903 [Main Thread] = INFO (CugRefresher.java:75) - 
-->2013-01-28 07:34:00,788 [Main Thread] = INFO (CugRefresher.java:154) 
-->2013-01-28 07:35:01,765 [Main Thread] = INFO (CugRefresher.java:164) 
-->2013-01-28 07:40:00,780 [Main Thread] = INFO (CugRefresher.java:194)

So i want to get data between 7:30 and 7:37 so it should return:

 
-->2013-01-28 07:34:00,788 [Main Thread] = INFO (CugRefresher.java:154) 
-->2013-01-28 07:35:01,765 [Main Thread] = INFO (CugRefresher.java:164)
4

I don't think what you are trying to do is possible using sed !

You have to perform date arithmetic here and fetch data between the specified range.

Here is a BASH script that might help you:

#!/bin/bash

date_1=$1
date_2=$2

e_date_1=$( date -d"$date_1" +"%s" )
e_date_2=$( date -d"$date_2" +"%s" )

while IFS=, read dt skip
do
        e_dt=$( date -d"$dt" +"%s" )
        if [ $e_dt -ge $e_date_1 ] && [ $e_dt -le $e_date_2 ]
        then
                printf "${dt},${skip}\n"
        fi
done < infile

Here is what I have in file: infile

$ cat infile
2013-01-28 07:33:57,903 [Main Thread] = INFO (CugRefresher.java:75)
2013-01-28 07:34:00,788 [Main Thread] = INFO (CugRefresher.java:154)
2013-01-28 07:35:01,765 [Main Thread] = INFO (CugRefresher.java:164)
2013-01-28 07:40:00,780 [Main Thread] = INFO (CugRefresher.java:194)

Here is the script output:

$ ./extract_bw_dates.sh "2013-01-28 07:30:00" "2013-01-28 07:37:00"
2013-01-28 07:33:57,903 [Main Thread] = INFO (CugRefresher.java:75)
2013-01-28 07:34:00,788 [Main Thread] = INFO (CugRefresher.java:154)
2013-01-28 07:35:01,765 [Main Thread] = INFO (CugRefresher.java:164)

I hope this helps.

5

This is simple to do with awk

awk '$0>=from && $0<=to' from="2013-01-28 07:30" to="2013-01-28 07:37" infile
or
awk '$0>="2013-01-28 07:30" && $0<="2013-01-28 07:37"' infile

2013-01-28 07:33:57,903 [Main Thread] = INFO (CugRefresher.java:75)
2013-01-28 07:34:00,788 [Main Thread] = INFO (CugRefresher.java:154)
2013-01-28 07:35:01,765 [Main Thread] = INFO (CugRefresher.java:164)

---------- Post updated at 09:34 ---------- Previous update was at 09:25 ----------

Here is an example to get last 4 hour of your log.

awk '$0>=from && $0<=to' from="$(date +%Y"-"%m"-"%d" "%H:%M -d -4hour)" to="$(date +%Y"-"%m"-"%d" "%H:%M)" infile
6

@Jotne How does this awk command will take care of the AM/PM in the timestamp for 12hr format?

---------- Post updated at 05:25 AM ---------- Previous update was at 04:10 AM ----------

The above awk solution wont work as i have to deal with different time formats as earlier mentioned and it the awk solution requires a fix time format

Like i said my time format can be anyhting like shown below:

 
<Jan 27, 2013 12:57:16 AM MST>
Jan 29, 2013 8:58:12 AM
2013-01-31 06:44:04,883

Pleae help...I am trying for something like AWK or SED or any regex combination

7

I think you need to convert your log to 24 hou r format before using the command.
Since I live in an area without AM/PM , I do not need to deal with this type of problem.
Here is some solution I use to convert log automatically

tail --follow=name /var/log/syslog | awk '!/found|cache/ {print | "tee /var/log/new.log"}' &

This remove all log entry with found & cache , buy you can convert to 24 hour format.
Then do all test on new.log

8

The log file can be anywhere between 10 Mb to 1 GB so i dont want to do that much amount of processing of converting 12hr format(AM/PM) to 24Hr in the log file itself

9

That's why I use the tail/awk combination to take out what I need to a new smaller file that I use to examine.