awk : collecting all data between two time frame

Shell Programming and Scripting
1

Hi Experts ,

I need your help to collect the complete data between two time frame from the log files, when I try awk it's collecting the data only which is printed with time stamp
for example, awk works well from "16:00 to 17:30" but its not collecting <line*> "from 17:30 to 18:00"

Requirement : The script will run every 15 mins from cron to collect the past 15 mins data then it will grep the error, the error will print between line1 to line8 also

Log file : 24 hours format
OS : Linux 6.2
Log file format:

2014-01-18 16:00:49,624 DEBUG [testhandler] XMLObjectType: S=xyz@http://www.xyz.org/2000/XMLDSchema
2014-01-18 16:10:49,624 DEBUG [testhandler] XMLObjectType: S=xyz@http://www.xyz.org/2000/XMLDSchema
2014-01-18 16:30:49,624 DEBUG [testhandler] XMLObjectType: S=xyz@http://www.xyz.org/2000/XMLDSchema
2014-01-18 16:50:49,624 DEBUG [testhandler] XMLObjectType: S=xyz@http://www.xyz.org/2000/XMLDSchema
2014-01-18 17:00:49,624 DEBUG [testhandler] XMLObjectType: S=xyz@http://www.xyz.org/2000/XMLDSchema
2014-01-18 17:30:49,624 DEBUG [testhandler] XMLObjectType: S=xyz@http://www.xyz.org/2000/XMLDSchema
<line1="http://G2.rc.xyz.org">
  <line2>
    <line3>
    <line5>
  </line6>
2014-01-18 17:50:49,624 DEBUG [test123]
User: data.production@test.com
<line1="http://G2.rc.xyz.org">
  <line2>
    <line3>
    <line5>
  </line6>
  <line7>
  <line8>
2014-01-18 18:00:49,624 DEBUG [testhandler] XMLObjectType: S=xyz@http://www.xyz.org/2000/XMLDSchema
2014-01-18 18:10:49,624 DEBUG [testhandler] XMLObjectType: S=xyz@http://www.xyz.org/2000/XMLDSchema
2 
sed -n '/17:30/,/18:00/p' file
3

what is expected output ?

4

balajesuri - I think sed will not help here, in case if the log error starting from "17:05" and script collecting the details from 17:30 then what is the output....

Akshay - The expected output

2014-01-18 17:30:49,624 DEBUG [testhandler] XMLObjectType: S=xyz@http://www.xyz.org/2000/XMLDSchema
<line1="http://G2.rc.xyz.org">
  <line2>
    <line3>
    <line5>
  </line6>
2014-01-18 17:50:49,624 DEBUG [test123]
User: data.production@test.com
<line1="http://G2.rc.xyz.org">
  <line2>
    <line3>
    <line5>
  </line6>
  <line7>
  <line8>
2014-01-18 18:00:49,624 DEBUG [testhandler] XMLObjectType: S=xyz@http://www.xyz.org/2000/XMLDSchema
5

I am not able to understand here.. If you're talking about why I hardcoded the time into the sed command, then I'd say that my post is merely a nudge to help you get started :slight_smile:

6

@zenkranti: balajesuri's sed is producing expected output.

7

@balajesuri,akshay, you can't use sed like that here, since it might just as well be that there is no log entry for 17:30 for example, but let's say 17:32 and then it won't work at all..

8

This would be fine, uses current row date

$ cat log
2014-01-18 16:00:49,624 DEBUG [testhandler] XMLObjectType: S=xyz@http://www.xyz.org/2000/XMLDSchema
2014-01-18 16:10:49,624 DEBUG [testhandler] XMLObjectType: S=xyz@http://www.xyz.org/2000/XMLDSchema
2014-01-18 16:30:49,624 DEBUG [testhandler] XMLObjectType: S=xyz@http://www.xyz.org/2000/XMLDSchema
2014-01-18 16:50:49,624 DEBUG [testhandler] XMLObjectType: S=xyz@http://www.xyz.org/2000/XMLDSchema
2014-01-18 17:00:49,624 DEBUG [testhandler] XMLObjectType: S=xyz@http://www.xyz.org/2000/XMLDSchema
2014-01-18 17:30:49,624 DEBUG [testhandler] XMLObjectType: S=xyz@http://www.xyz.org/2000/XMLDSchema
<line1="http://G2.rc.xyz.org">
  <line2>
    <line3>
    <line5>
  </line6>
2014-01-18 17:50:49,624 DEBUG [test123]
User: data.production@test.com
<line1="http://G2.rc.xyz.org">
  <line2>
    <line3>
    <line5>
  </line6>
  <line7>
  <line8>
2014-01-18 18:00:49,624 DEBUG [testhandler] XMLObjectType: S=xyz@http://www.xyz.org/2000/XMLDSchema
2014-01-18 18:10:49,624 DEBUG [testhandler] XMLObjectType: S=xyz@http://www.xyz.org/2000/XMLDSchema

awk -F, '
function dform(v){
                    gsub(/[-:]/," ",v)
                    return mktime(v)
                 }   
    dform($1)!=-1{
                    s = substr($1,1,10)" "start
                    e = substr($1,1,10)" "end 
                    f = dform($1)>=dform(s) && dform($1) <= dform(e) ? 1 : 0 
                 }f
         ' start="17:30:49" end="18:00:59" log

Resulting

2014-01-18 17:30:49,624 DEBUG [testhandler] XMLObjectType: S=xyz@http://www.xyz.org/2000/XMLDSchema
<line1="http://G2.rc.xyz.org">
  <line2>
    <line3>
    <line5>
  </line6>
2014-01-18 17:50:49,624 DEBUG [test123]
User: data.production@test.com
<line1="http://G2.rc.xyz.org">
  <line2>
    <line3>
    <line5>
  </line6>
  <line7>
  <line8>
2014-01-18 18:00:49,624 DEBUG [testhandler] XMLObjectType: S=xyz@http://www.xyz.org/2000/XMLDSchema

--edit---

In your requirement you didn't mention whether to consider date as well along with time or not

if date also you want to consider in query use following one, following will be helpful if log file is appended with date and time everyday.

awk -F, '
function dform(v){
                    gsub(/[-:]/," ",v)
                    return mktime(v)
                 }  
           NR==1 {
                    start = dform(start)
                     end  = dform(end)
                 }  
    dform($1)!=-1{     
                    f = dform($1)>=start && dform($1) <= end ? 1 : 0 
                 }f
        ' start="2014-01-18 17:30:49" end="2014-01-18 18:00:50" log
1 Like
9

You could something like this to get entries of the last 15 minutes:

perl -MPOSIX -lane '
  BEGIN{
    $c=strftime("%Y%m%d%H%M%S",localtime(time-15*60))
  }
  if(/^....-..-/) {
    $t=$F[0].$F[1];
    $t =~ s/[-:]|,.*//g
  }
  print if ($t ge $c) .. eof()
' file

or

perl -MPOSIX -slane '
  BEGIN{
    $c=strftime("%Y%m%d%H%M%S",localtime(time-$min*60))
  }
  if(/^....-..-/) {
    $t=$F[0].$F[1];
    $t =~ s/[-:]|,.*//g
  }
  print if ($t ge $c) .. eof()
' -- -min=15
1 Like