Collecting logs between two time stamps

Shell Programming and Scripting
1

Hi, please help me to collect the entire log files between two time stamp.

for example,
I am looking script to collect the entire log between "2015-03-27 15:59" to "2015-03-27 16:15" in the below sample log file.

OS : RHEL 6.3
Date/Time : 24 hours format, the time is printing each log file but the date won't print all the log entry. Also time printing starting and middle (after the date)

Sample logs:

15:59:07,091 INFO  [stdout] (serverlpool thread  -- 00) 2015-03-27 15:59:07,091:INFO   server design Pool -- 1066: bundles initialized                                 [system]: Console 
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-13) Creating Service {http://xyz.com/sche/claas/Acco
15:59:08,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-18) Creating Service {http://xyz.com/sche/wor/tt
15:59:0,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-1800) Creating Service {http://xyz.com/sche/claas/Add
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-2000) Creating Service {http://xyz.com/sche/cldaas/Acco
15:59,07 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-222) Creating Service {http://xyz.com/egg/asdd/ssco
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-133) Creating Service {http://xyz.com/sche/claas/Acco
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-1344) Creating Service {http://xyz.com/sche/claas/Acco
16:05:07,091 INFO  [stdout] (serverlpool thread  -- 00) 2015-03-27 16:05:07,091:INFO   server Thread Pool -- 86: bundles initialized                                 [system]: Console
16:05:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-13) Creating Service {http://xyz.com/sche/claas/Acco
16:05:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-18) Creating Service {http://xyz.com/sche/wor/tt
16:06:010,091 INFO  [stdout] (serverlpool thread  -- 00) 2015-03-27 16:06:10,091:INFO   server Thread Pool -- 86: bundles initialized                                 [system]: Console
aaaaa
bbbbb
cccccc
16:15:07,091 INFO  [stdout] (serverlpool thread  -- 00) 2015-03-27 16:15:07,091:INFO   server design Pool -- 106: bundles initialized

:

Appreciate your help.

2

Most log files have dates and times in monotonically non-decreasing order. Yours does not.

15:59:07
15:59:08
15:59:0
15:59:08
15:59:07
16:05:07
16:06:010
16:15:07

Most log files have timestamps in a consistent format. Yours does not.

15:59:0
15:59:08
16:06:010

Assuming that comma and space are field delimiters, is the date field on lines that contain dates always field 9?

Since timestamps aren't in order, how are we supposed to guess what date should apply to a line that does not have a date?

What is supposed to happen to log file lines that do not have a date or timestamp?

What output are you hoping to produce for the sample input you provided?

3

Don, Thanks for the reply.

Yes, the date is printing the same filed. I'm excepting the below output if I grep the time stamp from "2015-03-27 15:59" to "2015-03-27 16:05"

15:59:07,091 INFO  [stdout] (serverlpool thread  -- 00) 2015-03-27 15:59:07,091:INFO   server design Pool -- 1066: bundles initialized                                 [system]: Console 
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-13) Creating Service {http://xyz.com/sche/claas/Acco
15:59:08,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-18) Creating Service {http://xyz.com/sche/wor/tt
15:59:0,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-1800) Creating Service {http://xyz.com/sche/claas/Add
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-2000) Creating Service {http://xyz.com/sche/cldaas/Acco
15:59,07 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-222) Creating Service {http://xyz.com/egg/asdd/ssco
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-133) Creating Service {http://xyz.com/sche/claas/Acco
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-1344) Creating Service {http://xyz.com/sche/claas/Acco
16:05:07,091 INFO  [stdout] (serverlpool thread  -- 00) 2015-03-27 16:05:07,091:INFO   server Thread Pool -- 86: bundles initialized
4

Thank you for responding.
In my previous post, I asked four questions. You answered one of them.
If you choose not to answer the other three questions I asked in my previous post, I can't determine how to guess correctly at what you should be done.

And, for the record, greping won't work for what you're asking for. The grep utility uses regular expressions to look for certain patterns. It is not suitable for performing alphanumeric range comparisons.

5

Apologies, here is my reply for your questions...

Assuming that comma and space are field delimiters, is the date field on lines that contain dates always field 9?

Ans: The date's are printing always the same field.Sorry, is that 9 or 10?

12:18:17,734 INFO  [xyzout] (ajp-/xx.xx.xx.xxx:1234-1) 2015-03-30 12:18:17,733:INFO   :xyz-/xx.xx.xx.xxx:1234-1: smopBindingStub object in ABCDDataRetrievalVOg class
      [name_nname@email.domain.com.smb]: workingCreateQuotesFromabcdefData
12:20:17,734 INFO  [xyzout] (ajp-/xx.xx.xx.xxx:1234-21) 2015-03-30 12:20:17,733:INFO   :xyz-/xx.xx.xx.xxx:1234-1: smopBindingStub object in ABCDDataRetrievalVOg class
      [name_nname@email.domain.com.smb]: workingCreateQuotesFromabcdefData
12:25:17,734 INFO  [xyzout] (ajp-/xx.xx.xx.xxx:1234-21) 2015-03-30 12:25:17,733:INFO   :xyz-/xx.xx.xx.xxx:1234-1: smopBindingStub object in ABCDDataRetrievalVOg class
      [name_nname@email.domain.com.smb]: workingCreateQuotesFromabcdefData
12:26:17,734 INFO  [xyzout] (ajp-/xx.xx.xx.xxx:1234-22) 2015-03-30 12:26:17,733:INFO   :xyz-/xx.xx.xx.xxx:1234-1: smopBindingStub object in ABCDDataRetrievalVOg class
      [name_nname@email.domain.com.smb]: workingCreateQuotesFromabcdefData

Since timestamps aren't in order, how are we supposed to guess what date should apply to a line that does not have a date?

Ans: It seems like, the data is printing the first paragraph and time alone is printing the remaining lines so I can get all middle lines which doesn't have date if I collect the logs between two date and time.

What is supposed to happen to log file lines that do not have a date or timestamp?

Ans: Those are printing between two different date (or) timestamp. If that possible to collect all the logs lines between two different date or timestamp? For example, past 20 mins logs.

What output are you hoping to produce for the sample input you provided?

Ans:

15:59:07,091 INFO  [stdout] (serverlpool thread  -- 00) 2015-03-27 15:59:07,091:INFO   server design Pool -- 1066: bundles initialized                                 [system]: Console 
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-13) Creating Service {http://xyz.com/sche/claas/Acco
15:59:08,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-18) Creating Service {http://xyz.com/sche/wor/tt
15:59:0,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-1800) Creating Service {http://xyz.com/sche/claas/Add
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-2000) Creating Service {http://xyz.com/sche/cldaas/Acco
15:59,07 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-222) Creating Service {http://xyz.com/egg/asdd/ssco
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-133) Creating Service {http://xyz.com/sche/claas/Acco
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-1344) Creating Service {http://xyz.com/sche/claas/Acco
16:05:07,091 INFO  [stdout] (serverlpool thread  -- 00) 2015-03-27 16:05:07,091:INFO   server Thread Pool -- 86: bundles initialized

Also, can you please explain me the below script.


awk -F, '
function dform(v){
                    gsub(/[-:]/," ",v)
                    return mktime(v)
                 }
           NR==1 {
                    start = dform(start)
                     end  = dform(end)
                 }
    dform($1)!=-1{
                    f = dform($1)>=start && dform($1) <= end ? 1 : 0
                 }f
        ' start="$from" end="$to" $log1 > /script/xyz.txt
6

What has this

12:18:17,734 INFO  [xyzout] (ajp-/xx.xx.xx.xxx:1234-1) 2015-03-30 12:18:17,733:INFO . . . 
      [name_nname@email.domain.com.smb]: workingCreateQuotesFromabcdefData

to do with your problem resp. the sample given?

awk -F, '                                                               # run awk with field separator set to ","
function dform(v){                                                      # convert time string to binary representation
                    gsub(/[-:]/," ",v)                                  # replace "-" and ":" with " " in v
                    return mktime(v)                                    # do conversion
                 }
           NR==1 {                                                      # at start of file
                    start = dform(start)                                # convert start parameter
                     end  = dform(end)                                  # and end
                 }
    dform($1)!=-1{                                                      # if first field is a valid time string
                    f = dform($1)>=start && dform($1) <= end ? 1 : 0    # set "doprint" var "f" if $1 within boundaries   
                 }f                                                     # print line if "f"
        ' start="$from" end="$to" $log1 > /script/xyz.txt               # supply boundaries and log file; redirect

Be aware that mktime is not available in all awk versions.
You could use this snippet for your problem if you use [ ,] for the field separator and apply it to $10 instead of $1 . It would not print the last line in your desired output, though.

7

Thank you...

Yes, I gave him sample output to understand the date field.

please correct me, the below script not helping to get my excepted output...

#log
log1=/app/applicationdir/log/test.log

# Time stamp
to=`date "+%Y-%m-%d %H:%M:%S"`
from=`date "+%Y-%m-%d %H:%M:%S" -d "-20 minutes"`

#echo $from

awk -F, '
function dform(v){
                    gsub(/[-:]/," ",v)
                    return mktime(v)
                 }
           NR==1 {
                    start = dform(start)
                     end  = dform(end)
                 }
    dform($10)!=-1{
                    f = dform($10)>=start && dform($10) <= end ? 1 : 0
                 }f
        ' start="$from" end="$to" $log1 > /tmp/xyzlog.txt
8

It can't, as you

  • didn't use [ ,] as the field separator (add "*" to allow for multiple spaces between fields)
  • your to and from variables are date/time format, and $10 is time only. Use $9" "$10 in lieu of $10 alone.
9

Apologies , I am new to awk dform , Would be great if you modify the specific line or script and update here since I am getting error after changing the

 ($9,$10)

, Also not sure which line I need to replace for

[ ,]

, add "*" . Thanks!

10

I've done several sort of optimizing steps in one here. Try

awk -F"[ ,]*" '
function dform(v)       {gsub(/[-:]/," ",v)
                         return mktime(v)
                        }

NR==1                   {start = dform(start)
                         end   = dform(end)  
                        }

TMP=dform($9" "$10)!=-1 {f = (TMP>=start && TMP <= end)
                        }

f
 
' start="$from" end="$to"  $log1 > /tmp/xyzlog.txt
11

It's not working as excepted...

#more test.log

14:15:15,697 INFO  [stdout] (xyz-/xx.xx.xx.xxx:3009-4) 2015-03-31 14:15:15,697:INFO   :xyz-/xx.xx.xx.xxx:3009-4: Start new combine XRC ARC logic.....START
      [jerryknj@company.domain.com]: UtilCombineLine
14:18:15,697 INFO  [stdout] (xyz-/xx.xx.xx.xxx:3009-4) 2015-03-31 14:18:15,697:INFO   :xyz-/xx.xx.xx.xxx:3009-4: Start new combine XRC ARC logic.....START
      [jerryknj@company.domain.com]: UtilCombineLine
14:28:15,697 INFO  [stdout] (xyz-/xx.xx.xx.xxx:3009-4) 2015-03-31 14:28:15,697:INFO   :xyz-/xx.xx.xx.xxx:3009-4: Start new combine XRC ARC logic.....START
      [jerryknj@company.domain.com]: UtilCombineLine
14:28:15,697 INFO  [stdout] (xyz-/xx.xx.xx.xxx:3009-4) 2015-03-31 14:28:15,697:INFO   :xyz-/xx.xx.xx.xxx:3009-4: Start new combine XRC ARC logic.....START
      [jerryknj@company.domain.com]: UtilCombineLine
14:28:15,697 INFO  [stdout] (xyz-/xx.xx.xx.xxx:3009-4) 2015-03-31 14:28:15,697:INFO   :xyz-/xx.xx.xx.xxx:3009-4: Start new combine XRC ARC logic.....START
      [jerryknj@company.domain.com]: UtilCombineLine
14:28:15,697 INFO  [stdout] (xyz-/xx.xx.xx.xxx:3009-4) 2015-03-31 14:28:15,697:INFO   :xyz-/xx.xx.xx.xxx:3009-4: Start new combine XRC ARC logic.....START
      [jerryknj@company.domain.com]: UtilCombineLine

#sh -x test.sh or
#./test.sh


+ log1=/tmp/test.log
++ date '+%Y-%m-%d %H:%M:%S'
+ to='2015-03-31 14:30:34'
++ date '+%Y-%m-%d %H:%M:%S' -d '-30 minutes'
+ from='2015-03-31 14:00:34'
+ awk '-F[ ,]*' '
function dform(v)       {gsub(/[-:]/," ",v)
                         return mktime(v)
                        }

NR==1                   {start = dform(start)
                         end   = dform(end)
                        }

TMP=dform($9" "$10)!=-1 {f = (TMP>=start && TMP <= end)
                        }

f
' 'start=2015-03-31 14:00:34' 'end=2015-03-31 14:30:34' /tmp/test.log

#more /tmp/xyzlog.txt
empty.

12

Kidding? That input file is different in structure than the one post#1 and the output in post#3. With that input you can't expect an output close to what you desired.
And - No redirection - no output file, of course.

13

Apologies, I am messing here...

The post#1 and post#3 and post#11, all the outputs are taken from the same logs.

Just noticed, the logs date printing in different field in each samples... I think I have to work with development team to standardize the logs pattern.

Sample -1

12:00:45,026 INFO  [stdout] (pool-92-thread-5) 2015-03-31 12:00:45,026:ERROR  :pool-92-thread-1: [1420043245026] ContextTracking [system]: YCPContext
12:00:45,027 INFO  [stdout] (pool-92-thread-5) 2015-03-31 12:00:45,026:ERRORDTL:pool-92-thread-1: [1420003245526]java.lang.RuntimeException: ContextTracking

Sample - 2

14:15:15,697 INFO  [stdout] (xyz-/xx.xx.xx.xxx:3009-4) 2015-03-31 14:15:15,697:INFO   :xyz-/xx.xx.xx.xxx:3009-4: Start new combine XRC ARC logic.....START
      [jerryknj@company.domain.com]: UtilCombineLine
14:18:15,697 INFO  [stdout] (xyz-/xx.xx.xx.xxx:3009-4) 2015-03-31 14:18:15,697:INFO   :xyz-/xx.xx.xx.xxx:3009-4: Start new combine XRC ARC logic.....START
      [jerryknj@company.domain.com]: UtilCombineLine

Sample - 3

5:59:07,091 INFO  [stdout] (serverlpool thread  -- 00) 2015-03-27 15:59:07,091:INFO   server design Pool -- 1066: bundles initialized                                 [system]: Console 
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-13) Creating Service {http://xyz.com/sche/claas/Acco
15:59:08,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-18) Creating Service {http://xyz.com/sche/wor/tt
15:59:0,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-1800) Creating Service {http://xyz.com/sche/claas/Add
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-2000) Creating Service {http://xyz.com/sche/cldaas/Acco
15:59,07 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-222) Creating Service {http://xyz.com/egg/asdd/ssco
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-133) Creating Service {http://xyz.com/sche/claas/Acco
15:59:07,091 INFO [aaa.apache.xxx.service.factory.ReflectionBean] (ajp-/xx.xx.xx.xxx:port-1344) Creating Service {http://xyz.com/sche/claas/Acco
16:05:07,091 INFO  [stdout] (serverlpool thread  -- 00) 2015-03-27 16:05:07,091:INFO   server Thread Pool -- 86: bundles initialized

since I executed sh-x it's not showing the redirected file name, when I execute ksh -x or ./test.sh it showing and showing the update time stamp in the xyzlog.txt

Thanks for the help!