Combine logs span across multiple lines

Shell Programming and Scripting
1

Hi All,
I am having a log file

ERROR 2016-12-08 10:22:23.542 some data
**** some data******
**** some data******
**** some data******
DEBUG 2016-12-08 10:23:23.542 some data
**** some data******
**** some data******
**** some data******

when i grep the log file with

ERROR

am getting only the first line, but actual log line with respect to that timestamp spans across multiple lines.
Would like to know how we can combine the log lines in one line to get the entire line as a part of grep comand

2

IF you have "markers" and you know them all - DEBUG ERROR (you fill in the rest)

awk '  BEGIN {ok=0}
         { 
            if( index("|DEBUG|ERROR|", $1) ) 
            {
                 ok=( $1 == "ERROR")
            }
            if(ok) {print $0}
       } '  logfile

Note: this could be transformed into a one-liner awk statement. But since you seem not to know awk, I feel this is more helpful for understanding.

Change "|DEBUG|ERROR|" to include all of the "marker" words, the words you use to delimit your search.

1 Like
3

Hi.

An alternate solution using a (non-standard) member of the grep family:

cgrep +w <delimiter-string-list> <search-string> <filename>

will search <filename> for <search-string>. When found, it will also copy lines up to a line containing a match for <delimiter-string-list>.

Here are examples using an extended dataset:

#!/usr/bin/env bash

# @(#) s1       Demonstrate extraction of blocks of delimited text, cgrep.

# Utility functions: print-as-echo, print-line-with-visual-space, debug.
# export PATH="/usr/local/bin:/usr/bin:/bin"
LC_ALL=C ; LANG=C ; export LC_ALL LANG
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
em() { pe "$*" >&2 ; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C cgrep

FILE=${1-data1}

pl " Input data file $FILE:"
cat $FILE

markers='ERROR|DEBUG|CLARK|ZEPHYR'
search=CLARK
pl " Results, looking for $search:"
cgrep -D -E +I2 +w "$markers" $search $FILE

search=ERROR
pl " Results, looking for $search:"
cgrep -D -E +I2 +w "$markers" $search $FILE

search=DEBUG
pl " Results, looking for $search:"
cgrep -D -E +I2 +w "$markers" $search $FILE

exit 0

producing:

$ ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 3.16.0-4-amd64, x86_64
Distribution        : Debian 8.6 (jessie) 
bash GNU bash 4.3.30
cgrep ATT cgrep 8.15

-----
 Input data file data1:
ERROR 2016-12-08 10:22:23.542 some data
**** some data****** 1
**** some data****** 2
**** some data****** 3
DEBUG 2016-12-08 10:23:23.542 some data
**** some data****** 4
**** some data****** 5
**** some data****** 6
CLARK 2016-11-08 10:23:23.542 some data
**** some data****** 7
**** some data****** 8
**** some data****** 9
**** some data****** 9a
ZEPHYR 2015-11-08 10:23:23.542 some data
**** some data****** 10
**** some data****** 11
**** some data****** 12
DEBUG 2014-12-08 10:23:23.542 some data
**** some data****** 13
**** some data****** 14
**** some data****** 15

-----
 Results, looking for CLARK:
CLARK 2016-11-08 10:23:23.542 some data
**** some data****** 7
**** some data****** 8
**** some data****** 9
**** some data****** 9a

-----
 Results, looking for ERROR:
ERROR 2016-12-08 10:22:23.542 some data
**** some data****** 1
**** some data****** 2
**** some data****** 3

-----
 Results, looking for DEBUG:
DEBUG 2016-12-08 10:23:23.542 some data
**** some data****** 4
**** some data****** 5
**** some data****** 6
DEBUG 2014-12-08 10:23:23.542 some data
**** some data****** 13
**** some data****** 14
**** some data****** 15

The cgrep code needs to be acquired, compiled, and made available. I have done this several times over the years. An additional advantage is that cgrep is very fast.

Here are some details:

cgrep   shows context of matching patterns found in files (man)
Path    : ~/executable/cgrep
Version : 8.15
Type    : ELF 64-bit LSB executable, x86-64, version 1 (SYSV ...)
Home    : http://sourceforge.net/projects/cgrep/

Best wishes ... cheers, drl

1 Like
4 
awk '/^[[:upper:]]{2}/{f=0} /^ERROR/{f=1}f' file
1 Like