sed 2 hours in log

amazigh42 · February 10, 2013, 9:25am

Hi,

I want grep a log between two hours

cat server.log

sed -n "/24\/01\/2013 09:10/,/24\/01\/2013 10:45/p" server.log

24/01/2013 09:10
sssssssssssssss
cccccccccccccc
nnnnnnnnnnnnn
24/01/2013 10:10
uuuuuuuuuuuuuuu
jjjjjjjjjjjjjj
llllllllllllll
mmmmmmmmmmmmm
24/01/2013 10:30
oooooooooooo
sssssssssssss
qqqqqqqqqqq
24/01/2013 10:45

Here it's OK
but when the hour doesn't exist, it doesn't work

sed -n "/24\/01\/2013 09:11/,/24\/01\/2013 10:45/p" server.log

I would like when the begin hour doesn't exist, the script 'll take the first hour before else it'll take the begin of the log.

Idem with the end hour, if the hour doesn't exist, it 'll take the first hour after, else it'll take the end of the log.

For example

sed -n "/24\/01\/2013 09:05/,/24\/01\/2013 10:50/p" server.log

cat server.log

sed -n "/24\/01\/2013 10:15/,/24\/01\/2013 10:35/p" server.log

cat server.log

Thank you for your help.

Chubler_XL · February 10, 2013, 6:47pm

Don't think sed is capable of doing this, this awk script (nawk if you on Solaris) should do what you ask (just change the S and E variables to match your range):

awk -F"[/ ]" -vS="24/01/2013 09:05" -vE="24/01/2013 10:50" '
function dcmp(b) {
  if($3>b[3])return  1;
  if($3<b[3])return -1;
  if($2>b[2])return  1;
  if($2<b[2])return -1;
  if($1>b[1])return  1;
  if($1<b[1])return -1;
  if($4>b[4])return  1;
  if($4<b[4])return -1;
  return 0;
}
BEGIN{split(S, ds, "[/ ]"); split(E, de, "[/ ]") }
/[0-9]{2}\/[0-1][0-9]\/[[0-9]{4}/ {
   if(s&&dcmp(de)>=0) exit
   if(!s&&dcmp(ds)<=0) {f=x;w=1}
   if(!s&&dcmp(ds)>=0) {printf "%s",f; f=x; s=1 }
}
!w&&!s {f=f $0 "\n"}
s' server.log

MadeInGermany · February 11, 2013, 4:48am

Amazing!
Your code works with Posix-compatible GNU awk and HP-UX awk.
Somehow Solaris nawk does not work (prints nothing),
and Solaris Posix /usr/xpg4/bin/awk wants correct arguments:
-v S="..." -v E="..."

amazigh42 · February 11, 2013, 7:55am

chubler_xl:

Don't think sed is capable of doing this, this awk script (nawk if you on Solaris) should do what you ask (just change the S and E variables to match your range):

awk -F"[/ ]" -vS="24/01/2013 09:05" -vE="24/01/2013 10:50" '
function dcmp(b) {
  if($3>b[3])return  1;
  if($3<b[3])return -1;
  if($2>b[2])return  1;
  if($2<b[2])return -1;
  if($1>b[1])return  1;
  if($1<b[1])return -1;
  if($4>b[4])return  1;
  if($4<b[4])return -1;
  return 0;
}
BEGIN{split(S, ds, "[/ ]"); split(E, de, "[/ ]") }
/[0-9]{2}\/[0-1][0-9]\/[[0-9]{4}/ {
   if(s&&dcmp(de)>=0) exit
   if(!s&&dcmp(ds)<=0) {f=x;w=1}
   if(!s&&dcmp(ds)>=0) {printf "%s",f; f=x; s=1 }
}
!w&&!s {f=f $0 "\n"}
s' server.log

Thanks a lot for your help.
I launch the script but there is no result or message ?
My shell is bash.

MadeInGermany · February 11, 2013, 8:32am

Any sh-compatible shell will do it, because it does nothing than passing a multi-line 'string' to awk. But the awk version matters!
Surprisingly, Solaris nawk does not handle expr{dig,dig} .
Here comes a version that is understood by nawk and /usr/xpg4/bin/awk:

awk -F"[/ ]" -v S="24/01/2013 09:05" -v E="24/01/2013 10:50" '
function dcmp(b) {
  if($3>b[3])return  1;
  if($3<b[3])return -1;
  if($2>b[2])return  1;
  if($2<b[2])return -1;
  if($1>b[1])return  1;
  if($1<b[1])return -1;
  if($4>b[4])return  1;
  if($4<b[4])return -1;
  return 0;
}
BEGIN{split(S, ds, "[/ ]"); split(E, de, "[/ ]") }
/^[0-9][0-9]\/[0-1][0-9]\/[0-9][0-9][0-9][0-9] / {
   if(s&&dcmp(de)>=0) exit
   if(!s&&dcmp(ds)<=0) {f=x;w=1}
   if(!s&&dcmp(ds)>=0) {printf "%s",f; f=x; s=1 }
}
!w&&!s {f=f $0 "\n"}
s' server.log

amazigh42 · February 11, 2013, 11:29am

Thanks a lot, it works.

But there is a little problem.

The result is

The last line is missing

cat server.log

error jonas
aaaaaaaaaaaaaa
bbbbbbbbbbbbb
cccccccccccc
24/01/2013 09:10
sssssssssssssss
cccccccccccccc
nnnnnnnnnnnnn
24/01/2013 10:10
uuuuuuuuuuuuuuu
jjjjjjjjjjjjjj
llllllllllllll
mmmmmmmmmmmmm
24/01/2013 10:30
oooooooooooo
sssssssssssss
qqqqqqqqqqq
24/01/2013 10:45
vvvvvvvvv
sssssssss
wwwwwwwwww

awk -F"[/ ]" -v S="24/01/2013 10:10" -v E="24/01/2013 10:30" '
function dcmp(b) {
  if($3>b[3])return  1;
  if($3<b[3])return -1;
  if($2>b[2])return  1;
  if($2<b[2])return -1;
  if($1>b[1])return  1;
  if($1<b[1])return -1;
  if($4>b[4])return  1;
  if($4<b[4])return -1;
  return 0;
}
BEGIN{split(S, ds, "[/ ]"); split(E, de, "[/ ]") }
/^[0-9][0-9]\/[0-1][0-9]\/[0-9][0-9][0-9][0-9] / {
   if(s&&dcmp(de)>=0) exit
   if(!s&&dcmp(ds)<=0) {f=x;w=1}
   if(!s&&dcmp(ds)>=0) {printf "%s",f; f=x; s=1 }
}
!w&&!s {f=f $0 "\n"}
s' server.log

Chubler_XL · February 11, 2013, 3:39pm

Thanks for your Solaris corrections MadeInGermany, nice to know that {n,n} regs don't work with nawk.

I'm not quite sure what the requirement is around this last like, the following change
will include the final date line that matches/exceeds the Ending date/time

Replace:

if(s&&dcmp(de)>=0) exit

With:

if(s&&dcmp(de)>=0) {print; exit}

amazigh42 · February 12, 2013, 2:37am

chubler_xl:

Thanks for your Solaris corrections MadeInGermany, nice to know that {n,n} regs don't work with nawk.

I'm not quite sure what the requirement is around this last like, the following change
will include the final date line that matches/exceeds the Ending date/time

Replace:
if(s&&dcmp(de)>=0) exit
With:
if(s&&dcmp(de)>=0) {print; exit}

That's great.
Thanks to Chubler_XL and MadeInGermany for this superb bit of script.
I'm a beginner, i can not understand some lines, can you help me decipher the blue lines.

awk -F"[/ ]" -v S="24/01/2013 10:10" -v E="24/01/2013 10:30" '
function dcmp(b) {
  if($3>b[3])return  1;
  if($3<b[3])return -1;
  if($2>b[2])return  1;
  if($2<b[2])return -1;
  if($1>b[1])return  1;
  if($1<b[1])return -1;
  if($4>b[4])return  1;
  if($4<b[4])return -1;
  return 0;
}
BEGIN{split(S, ds, "[/ ]"); split(E, de, "[/ ]") }
/^[0-9][0-9]\/[0-1][0-9]\/[0-9][0-9][0-9][0-9] / {
   if(s&&dcmp(de)>=0) {print; exit}
   if(!s&&dcmp(ds)<=0) {f=x;w=1}
   if(!s&&dcmp(ds)>=0) {printf "%s",f; f=x; s=1 }
}
!w&&!s {f=f $0 "\n"}
s' server.log

Chubler_XL · February 12, 2013, 3:55am

if($3>b[3])return 1;
if($3,b[3])return -1;
...
if 3rd field (year) of current line is greater than 3rd field in input parameter (year of start or end date) return 1 (ie > 0).
As you can see years (field 3) are compared first and only if the two year values are the same are month values (filed 2) compared, then days (filed 1) and finally time (field 4).
The final result is that dcmp() returns 0 if dates are identical,1 if current line is greater than input param, and -1 if current line is less than input param.

/^[1][0-9]\/[0-1][0-9]\/[0-9][0-9][0-9][0-9] /
Match lines that start with two digits followed by slash then 0 or 1 then digit then slash and finally 4 digits (ie line starts with a 99/99/9999 format date).

if\(s&&dcmp\(de\)&gt;=0\) \{print; exit\}

if start date already found (s is non blank/zero) and date in current line >= date-end then print current line and stop processing.

if\(!s&&dcmp\(ds\)&lt;=0\) \{f=x;w=1\}

if start date not found yet (s is not set) and date in current line <= date-start, blank contents of var f and set w flag. f is a buffer that contains all lines found from start of document.

if\(!s&&dcmp\(ds\)&gt;=0\) \{printf "%s",f; f=x; s=1 \}

if start date not found yet (s is not set) and date in current line >= date-start, print f buffer, blank it and set start date found flag (s=1).

!w&&!s {f=f $0 "\n"}
if w and s flags are not set append current line to the f buffer

s
if s flag is set print current line

0-9 ↩︎

amazigh42 · February 12, 2013, 8:11am

What an limpid explaination ! It's clearer for me, thank you.

I have a particular case, when i have a gzip file like server.log.gz
I tried to use zcat in the script.
An error has occured.
Fatal can't open zcat server.log.gz
What's the problem ?

awk -F"[/ ]" -v S="24/01/2013 10:10" -v E="24/01/2013 10:30" '
function dcmp(b) {
  if($3>b[3])return  1;
  if($3<b[3])return -1;
  if($2>b[2])return  1;
  if($2<b[2])return -1;
  if($1>b[1])return  1;
  if($1<b[1])return -1;
  if($4>b[4])return  1;
  if($4<b[4])return -1;
  return 0;
}
BEGIN{split(S, ds, "[/ ]"); split(E, de, "[/ ]") }
/^[0-9][0-9]\/[0-1][0-9]\/[0-9][0-9][0-9][0-9] / {
   if(s&&dcmp(de)>=0) {print; exit}
   if(!s&&dcmp(ds)<=0) {f=x;w=1}
   if(!s&&dcmp(ds)>=0) {printf "%s",f; f=x; s=1 }
}
!w&&!s {f=f $0 "\n"}
s' "zcat server.log.gz"

Chubler_XL · February 12, 2013, 1:40pm

If awk has no file parameter it will process stdin so pipe output of zcat to awk like this:

 zcat server.log.gz | awk -F"[/ ]" -v S="24/01/2013 10:10" -v E="24/01/2013 10:30" '...code here...'

MadeInGermany · February 12, 2013, 1:48pm

On traditional Unix zcat is linked to the traditional uncompress rather than gunzip.
In this case replace zcat with gunzip -c .

amazigh42 · February 12, 2013, 3:18pm

Here no problems, just solutions.
Thanks the guys.
I would like to modify the script.
I want to use either cat or gunzip following if it is a single file or gzip file.
Have you ideas about conditions in red because it didn't work ?

file=$1
if [ -f *.gz] 
     then $command="gunzip -c" 
     else $command=cat
fi
$command $file | awk -F"[/ ]" -v S="24/01/2013 10:10" -v E="24/01/2013 10:30" '
function dcmp(b) {
  if($3>b[3])return  1;
  if($3<b[3])return -1;
  if($2>b[2])return  1;
  if($2<b[2])return -1;
  if($1>b[1])return  1;
  if($1<b[1])return -1;
  if($4>b[4])return  1;
  if($4<b[4])return -1;
  return 0;
}
BEGIN{split(S, ds, "[/ ]"); split(E, de, "[/ ]") }
/^[0-9][0-9]\/[0-1][0-9]\/[0-9][0-9][0-9][0-9] / {
   if(s&&dcmp(de)>=0) {print; exit}
   if(!s&&dcmp(ds)<=0) {f=x;w=1}
   if(!s&&dcmp(ds)>=0) {printf "%s",f; f=x; s=1 }
}
!w&&!s {f=f $0 "\n"}
s

MadeInGermany · February 12, 2013, 3:42pm

if [ -f $file.gz ] 
     then command="gunzip -c" 
     else command=cat
fi

You do want to test for $file.gz, don't you.
The shell [ ] wants a space on both sides. If you like you can rephrase it

if test -f $file.gz

The shell uses var= in assignment, and $var in reference. (While perl uses $var in both assignment and reference.)

amazigh42 · February 13, 2013, 12:45am

Good morning,

It's OK

I began to be embarrassed to ask questions constantly but :
1 - how redirect the output to another file instead of standard output.
2 - Sorry, I forgot the bracket in the pattern 24/01/2013 10:10 which becomes [24/01/2013 10:10]
I try things but the cure is worse than the problem itself.
Thanks in advance for yours invaluable help.

vi test

file=$1
if [ -f $file.gz ]
     then command="gunzip -c"
     else command=cat
fi
$command $file | awk -F"[/ ]" -v S="24/01/2013 10:10" -v E="24/01/2013 10:30" '
function dcmp(b) {
  if($3>b[3])return  1;
  if($3<b[3])return -1;
  if($2>b[2])return  1;
  if($2<b[2])return -1;
  if($1>b[1])return  1;
  if($1<b[1])return -1;
  if($4>b[4])return  1;
  if($4<b[4])return -1;
  return 0;
}
BEGIN{split(S, ds, "[/ ]"); split(E, de, "[/ ]") }
/^[0-9][0-9]\/[0-1][0-9]\/[0-9][0-9][0-9][0-9] / {
   if(s&&dcmp(de)>=0) {print; exit}
   if(!s&&dcmp(ds)<=0) {f=x;w=1}
   if(!s&&dcmp(ds)>=0) {printf "%s",f; f=x; s=1 }
}
!w&&!s {f=f $0 "\n"}
s

.
.
.

MadeInGermany · February 13, 2013, 5:09am

You miss the closing ' at the very end!
An exercise:

echo '
this
is
a
multiline
string'

And

echo '
line1
line2' > newfile

amazigh42 · February 13, 2013, 7:16am

Thanks MadeInGermany for your reply to my first question, can you answer my second question above because i'm drowned In fact there is bracket around the date [24/01/2013 10:10]

MadeInGermany · February 14, 2013, 7:52am

file=$1
if [ -f $file.gz ]
     then command="gunzip -c"
     else command=cat
fi
$command $file | awk -F"[/ ]" -v S="24/01/2013 10:10" -v E="24/01/2013 10:30" '
function dcmp(b) {
  if($3>b[3])return  1;
  if($3<b[3])return -1;
  if($2>b[2])return  1;
  if($2<b[2])return -1;
  if($1>b[1])return  1;
  if($1<b[1])return -1;
  if($4>b[4])return  1;
  if($4<b[4])return -1;
  return 0;
}
BEGIN { split(S, ds, "[/ ]"); split(E, de, "[/ ]") }
/^\[?[0-9][0-9]\/[0-1][0-9]\/[0-9][0-9][0-9][0-9] / {
   sub("^\[",""); sub("\]$","");
   if(s&&dcmp(de)>=0) {print; exit}
   if(!s&&dcmp(ds)<=0) {f=x;w=1}
   if(!s&&dcmp(ds)>=0) {printf "%s",f; f=x; s=1 }
}
!w&&!s {f=f $0 "\n"}
s'

The awk 'script' is modified: ^\[? in the implicit if clause accepts a [ at the beginning of the line. The sub statements strip off a leading [ and a trailing ] so the rest of the awk script should work as before.

amazigh42 · February 15, 2013, 2:06pm

madeingermany:

file=$1
if [ -f $file.gz ]
   then command="gunzip -c"
   else command=cat
fi
$command $file | awk -F"[/ ]" -v S="24/01/2013 10:12" -v E="24/01/2013 10:28" '
function dcmp(b) {
  if($3>b[3])return  1;
  if($3<b[3])return -1;
  if($2>b[2])return  1;
  if($2<b[2])return -1;
  if($1>b[1])return  1;
  if($1<b[1])return -1;
  if($4>b[4])return  1;
  if($4<b[4])return -1;
  return 0;
}
BEGIN { split(S, ds, "[/ ]"); split(E, de, "[/ ]") }
/^\[?[0-9][0-9]\/[0-1][0-9]\/[0-9][0-9][0-9][0-9] / {
   sub("^\[",""); sub("\]$","");
   if(s&&dcmp(de)>=0) {print; exit}
   if(!s&&dcmp(ds)<=0) {f=x;w=1}
   if(!s&&dcmp(ds)>=0) {printf "%s",f; f=x; s=1 }
}
!w&&!s {f=f $0 "\n"}
s'

The awk 'script' is modified: ^\[? in the implicit if clause accepts a [ at the beginning of the line. The sub statements strip off a leading [ and a trailing ] so the rest of the awk script should work as before.

Hello,
It doesn't work with the above modifications
The script returns errors code.

I remember that with the initial conditions and these hours (10:12 and 10:28), i should have this display

[24/01/2013 10:10]
uuuuuuuuuuuuuuu
jjjjjjjjjjjjjj
llllllllllllll
mmmmmmmmmmmmm
[24/01/2013 10:30]

With these hours (10:30 and 10:47), i should have this display

[24/01/2013 10:30]
oooooooooooo
sssssssssssss
qqqqqqqqqqq
[24/01/2013 10:45]
vvvvvvvvv
sssssssss
wwwwwwwwww

With these hours (09:05 and 10:15), i should have this[COLOR=Sienna][COLOR=Red]display

error jonas
aaaaaaaaaaaaaa
bbbbbbbbbbbbb
cccccccccccc
[24/01/2013 09:10]

I catch my breath.
Thank you for all.
[/COLOR][/COLOR]

./my_script server.log

The input file is

 cat server.log

error jonas
aaaaaaaaaaaaaa
bbbbbbbbbbbbb
cccccccccccc
[24/01/2013 09:10]
sssssssssssssss
cccccccccccccc
nnnnnnnnnnnnn
[24/01/2013 10:10]
uuuuuuuuuuuuuuu
jjjjjjjjjjjjjj
llllllllllllll
mmmmmmmmmmmmm
[24/01/2013 10:30]
oooooooooooo
sssssssssssss
qqqqqqqqqqq
[24/01/2013 10:45]
vvvvvvvvv
sssssssss
wwwwwwwwww

Chubler_XL · February 17, 2013, 4:15pm

This should correct the "[" and "]" chars around the data and time line.

file=$1
if [ -f $file.gz ]
     then command="gunzip -c"
     else command=cat
fi
$command $file | awk -F"[/ \\\][]" -vS="24/01/2013 10:12" -vE="24/01/2013 10:28" '
function dcmp(b) {
  if($4>b[3])return  1;
  if($4<b[3])return -1;
  if($3>b[2])return  1;
  if($3<b[2])return -1;
  if($2>b[1])return  1;
  if($2<b[1])return -1;
  if($5>b[4])return  1;
  if($5<b[4])return -1;
  return 0;
}
BEGIN{split(S, ds, "[/ ]"); split(E, de, "[/ ]") }
/^[[][0-9][0-9]\/[0-1][0-9]\/[[0-9][0-9][0-9][0-9] / {
   if(s&&dcmp(de)>=0) {print; exit}
   if(!s&&dcmp(ds)<=0) {f=x;w=1}
   if(!s&&dcmp(ds)>=0) {printf "%s",f; f=x; s=1 }
}
!w&&!s {f=f $0 "\n"}
s'

Can you explain your start+stop times criteria some more:

hours (10:12 and 10:28) Why does this start printing and 10:10, isn't that before the start time?

hours (09:05 and 10:15) why does it stop at 09:10?