Calculate total of log by hour

Shell Programming and Scripting
1

Hi,

Just wondering, is there anyway I can get the total of logs generated by hours ? Let say I have these logs,

Sep 23 04:48:43 hsbcufs: [ID 213553 kern.notice] NOTICE: realloccg /: file system full
Sep 23 04:48:47 hsbcufs: [ID 845546 kern.notice] NOTICE: alloc: /: file system full
Sep 23 04:48:51 hsbcufs: [ID 213553 kern.notice] NOTICE: realloccg /: file system full
Sep 23 04:48:55 hsbclast message repeated 1 time
Sep 23 04:48:59 hsbcufs: [ID 845546 kern.notice] NOTICE: alloc: /: file system full
Sep 23 04:50:12 hsbclast message repeated 19 times
Sep 23 04:50:16 hsbcufs: [ID 213553 kern.notice] NOTICE: realloccg /: file system full
Sep 23 04:50:20 hsbcufs: [ID 845546 kern.notice] NOTICE: alloc: /: file system full
Sep 23 04:52:22 hsbclast message repeated 30 times
Sep 23 04:52:26 hsbcufs: [ID 213553 kern.notice] NOTICE: realloccg /: file system full
Sep 23 04:52:30 hsbcufs: [ID 845546 kern.notice] NOTICE: alloc: /: file system full
Sep 23 04:53:22 hsbclast message repeated 13 times
Sep 23 04:53:26 hsbcufs: [ID 213553 kern.notice] NOTICE: realloccg /: file system full
Sep 23 04:53:30 hsbcufs: [ID 845546 kern.notice] NOTICE: alloc: /: file system full
Sep 23 04:55:03 hsbclast message repeated 23 times
Sep 23 04:55:07 hsbcufs: [ID 213553 kern.notice] NOTICE: realloccg /: file system full
Sep 23 04:55:11 hsbcufs: [ID 845546 kern.notice] NOTICE: alloc: /: file system full
Sep 23 04:55:40 hsbclast message repeated 7 times
Sep 23 04:55:44 hsbcufs: [ID 845546 kern.notice] NOTICE: alloc: /: file system full
Sep 23 04:56:44 hsbclast message repeated 15 times
Sep 23 04:56:48 hsbcufs: [ID 213553 kern.notice] NOTICE: realloccg /: file system full
Sep 23 04:56:52 hsbcufs: [ID 845546 kern.notice] NOTICE: alloc: /: file system full
Sep 23 05:00:51 hsbclast message repeated 59 times
Sep 23 05:00:55 hsbcufs: [ID 213553 kern.notice] NOTICE: realloccg /: file system full
Sep 23 05:01:03 hsbclast message repeated 2 times
Sep 23 05:01:07 hsbcufs: [ID 845546 kern.notice] NOTICE: alloc: /: file system full
Sep 23 05:02:16 hsbclast message repeated 17 times
Sep 23 05:02:20 hsbcufs: [ID 213553 kern.notice] NOTICE: realloccg /: file system full
Sep 23 05:02:24 hsbcufs: [ID 845546 kern.notice] NOTICE: alloc: /: file system full
Sep 23 05:02:52 hsbclast message repeated 7 times
Sep 23 05:02:56 hsbcufs: [ID 213553 kern.notice] NOTICE: realloccg /: file system full
Sep 23 05:03:00 hsbcufs: [ID 845546 kern.notice] NOTICE: alloc: /: file system full
Sep 23 05:03:16 hsbclast message repeated 4 times
Sep 23 05:03:20 hsbcufs: [ID 213553 kern.notice] NOTICE: realloccg /: file system full
Sep 23 05:03:24 hsbcufs: [ID 845546 kern.notice] NOTICE: alloc: /: file system full
Sep 23 05:07:27 hsbclast message repeated 60 times
Sep 23 05:07:31 hsbcufs: [ID 213553 kern.notice] NOTICE: realloccg /: file system full
Sep 23 05:07:35 hsbcufs: [ID 845546 kern.notice] NOTICE: alloc: /: file system full
Sep 23 05:08:56 hsbclast message repeated 20 times
Sep 23 05:09:00 hsbcufs: [ID 845546 kern.notice] NOTICE: alloc: /: file system full
Sep 23 05:13:59 hsbclast message repeated 74 times
Sep 23 05:14:03 hsbcufs: [ID 213553 kern.notice] NOTICE: realloccg /: file system full
Sep 23 05:14:07 hsbcufs: [ID 845546 kern.notice] NOTICE: alloc: /: file system full
Sep 23 05:15:20 hsbclast message repeated 18 times
Sep 23 05:15:24 hsbcufs: [ID 213553 kern.notice] NOTICE: realloccg /: file system full

I would like to get the stat like ,

Hours Total line
04 ______52
05 ______ 41
06 ____ 65
07 ____ 35

and get them in minutes as well ?

Many thanks everyone !!!

2 
$ awk -F"[ :]" '{print $3}' log.log  | sort | uniq -c
     22 04
     23 05
1 Like
3

you will need to write a script to parse the log.

1 Like
4 
$ awk -F"[ :]" '{a[$3]++;next}END{for(i in a){print i"_--->"a}}' log.log 
04_--->22
05_--->23
1 Like
5

thanks, but the code doesn't seems to work though, i get total of line instead

adm@hsbc> awk -F"[ :]" '{a[$3]++;next}END{for(i in a){print i"_--->"a}}' messages
_--->274

adm@hsbc> awk -F"[ :]" '{print $3}' messages  | sort | uniq -c
 274
6 
$ awk -F"[ :]" '{a[$3]++;next}END{for(i in a){print i"_--->"a}}' log.log 
04_--->2
05_--->2

$ cat log.log
Sep 23 04:56:48 hsbcufs: [ID 213553 kern.notice] NOTICE: realloccg /: file system full
Sep 23 04:56:52 hsbcufs: [ID 845546 kern.notice] NOTICE: alloc: /: file system full
Sep 23 05:00:51 hsbclast message repeated 59 times
Sep 23 05:00:55 hsbcufs: [ID 213553 kern.notice] NOTICE: realloccg /: file system full 

$ awk -F"[ :]" '{print $3}' log.log  | sort | uniq -c
      2 04
      2 05

for me its working fine. can you post the original message contents

---------- Post updated at 07:35 PM ---------- Previous update was at 07:35 PM ----------

if you are in solaris, please use nawk

1 Like
7

Thanks, I used nawk, and the output that I have is 00 till 59, looks like a minute count ?

nawk -F"[ :]" '{print $3}' withdrawal.log  | sort | uniq -c

11340 00
10371 01
9771 02
9869 03
10782 04
12424 05
14427 06
19253 07
17638 08
16186 09
17690 10
18769 11
14854 12
10323 13
10547 14
10116 15
10626 16
10397 17
10603 18
10622 19
10541 20
9801 21
10152 22
10409 23
10288 24
9778 25
10181 26
10013 27
11081 28
9419 29
10873 30
10039 31
10041 32
9730 33
10446 34
10695 35
10562 36
10799 37
10754 38
10730 39
10573 40
10432 41
10470 42
10877 43
10152 44
10661 45
10315 46
11294 47
9727 48
9937 49
9980 50
10526 51
10148 52
9858 53
10091 54
10298 55
10307 56
10153 57
10481 58
11003 59
8

did u give space between [ and :

 
-F"[ :]"
1 Like
9

Hi,

Apologies in advance, the cmd works for /var/adm/messages, but in the other log(abit sensitive to show)

adm@hsbc1> nawk -F"[ :]" '{print $3}' log.log | sort | uniq -c

  24 11
  18 12
  52 13
  56 14
  30 15
  39 16
  46 17
 178 18
 329 19
  47 20
  14 21
  16 22
  22 23
  24 24
  28 25
  35 26
  71 27
  10 28
   9 29
   4 30
   5 31
  23 32
  22 33
  24 34
  79 35
  20 36
  14 37
  12 38
  40 39
  43 40
  42 41
  24 42
  18 43
   5 44
   4 45
   4 46
   1 47
   4 48
  32 49
  96 50
  21 51
  17 52
  38 53
  34 54
  36 55
10 
 
awk -F"[ :]" '{print $2}' log.log  | sort | uniq -c
1 Like
11

oh, that was easy :smiley: ...thanks very much, can you explain what is 3 or 2 that you changed ?

12

-F"[ :]" --> you are splits the entire line by space and : as delimeter.

In the below line

2011-09-26 03:11:58,015 Thread-20565591 HTTP op [/createtemplatename

$1 --> 2011-09-26
$2 --> 03
$3 --> 11
$4 --> 58,015 and so.. on
$5 --> Thread-20565591

1 Like
13

superb!! many thanks !!!

14

Hi again,

Now I have the list of total count by hours, is there any possibilities to extract the MAX from the total count, and count them in minutes ?

Something like,

10342 00
51300 01
82801 02
2900 03
3306 04
4148 05
5560 06
16882 07
396 08
4608 09
2686 10
8635 11
8798 12
8402 13
763 14
805 15
930 16
1735 17
1575 18
954 19
805 20
605 21
301 22
250 23

Hour 02 is the higher, then I want to extract this hour and count by minutes on this hour ?

15 
awk -F"[ :]" '{print $2}' log.log  | sort | uniq -c > hour.log
high_hour=$(sort -nr hour.log | head -1 | cut -d" " -f2)
awk -F"[ :]" -v h=$high_hour '$2==h {print $3}' log.log | sort | uniq -c > min.log
1 Like