HELP: Shell Script to read a Log file line by line and extract Info based on KEYWORDS matching

Shell Programming and Scripting
1

I have a LOG file which looks like this

 
Import started at: Mon Jul 23 02:13:01 EDT 2012
Initialization completed in 2.146 seconds.
--------------------------------------------------------------------------------
--
Import summary for Import item: PolicyInformation
--------------------------------------------------------------------------------
--
Batch size: 1.
Commit count: 100.
Error Tolerance Level: 1.
Error Count: 0.
Amount of data processed: 10.
Amount of objects committed: 8.
Data Importer initialization time: 1 seconds.
Data Importer execution began: Mon Jul 23 02:13:03 EDT 2012
Data Importer execution ended: Mon Jul 23 02:13:06 EDT 2012
Data Importer completed in 2.735 seconds.
Total flush time: 0 seconds.
Total commit time: 0.008 seconds.
Affected tables (4):
Table name: PolicyHdr, Affected number of rows: 118.
Table name: PolicyDetl, Affected number of rows: 76.
Table name: ClaimSeq, Affected number of rows: 211.
Table name: Claimant, Affected number of rows: 116.
----------------------------------------------------------------------------------
 
Program exiting with exit code: 0.
Import completed successfully with no errors.
Import ended at: Mon Jul 23 02:13:06 EDT 2012
Import completed in 4.891 seconds.

I have KEYWORDS whcih are these
Error Count:
Amount of data processed:
Import completed in
and many more

Whats the best way to open and loop thru the file line by line looking for Keywords like these and extracting rest of the information .

i've looked/read several posts but nothing quiet what i need.
i tried GREP/AWK & SED but nothing is giving me what i want... (obviously im not an expert scriptor)

now this is what i've cooked up so far

#!/bin/bash
file="/home/myuser/importlog.txt"
val1='Import completed in'
while IFS= read -r line
do
let count++ # display $line or do somthing with $line
#echo "$count $line"
fit=`expr substr "$line" 1 17`
if [ $fit == $val1 ]
then
echo "Found IT $line"
fi
done <"$file"

it fails with line 9: [: too many arguments

not sure why its failing when matching the $fit == $val1 coz at command line it works

$ line='Import completed in'
$ fit=`expr substr "$line" 1 17`
$ echo $fit
Import completed

can any of you experts quickly cut my agony by helping me out... sincerely appreciated....

2

Start with this:

Write all of your keywords or phrases to a file, call it key.txt.

grep -Ff key.txt your_logfile_name

Beware of captialization and spaces, grep will do an EXACT match on your keywords.
And print the whole line.

3

Thanks Jim,
but i want the control inside the loop so i can do other operations like Substring on FOUND Lines
OR
look for additional info based on what keyword it found....

so i was looking something like this..

case Keyword1
{
do something
}
case Keyword2
{
do onething
do twothing
}

4

For case, you can always use case...

while read STRING
do
case "$STRING" in
*abc*)
        stuff
        ;;

*qwerty*)
        other stuff
        ;;

abcde)
        yet more stuff
        ;;
esac
done < inputfile
1 Like
5

i tried 2nd CASE statement

#!/bin/bash
file="/home/mbuser/log.txt"
val1='Load completed in'
fit=holy
err=crap
while IFS= read -r line
do
        let count++ # display $line or do somthing with $line
        #echo "$count $line"
case "$line" in
'*Import completed*')
        echo "Completed"
        fit=`expr substr "$line" 21 7`
        ;;
'*Error Count:*')
        echo "ERRROR"
        err=`expr substr "$line" 13 7`
        ;;
esac
done <"$file"
echo $fit $err

it outputs
holy crap ==< which i have initialized at top....as if it didnt go in the case statement.. what am i missing?

6

I don't think those strings in the ) need to be in single quotes.

7

if i dont put single QUOTES it fails with

./krun: line 11: syntax error near unexpected token `Count:*'
./krun: line 11: `Error Count:)'

Double Quotes doesnt make any difference.... same behaviou as Single Quote

8

You only need to quote the space in the pattern (a backslash would do it).

Regards,
Alister

9

:wall:

---------- Post updated at 11:44 PM ---------- Previous update was at 11:40 PM ----------