Another method for this snippet

Shell Programming and Scripting
1

Hi All, i believe this is not very efficient. another method would be appreciated for these. basically i read a file with tab delimited column and pass the column to another perl script.

while read line
do
        timestamp=`echo "$line"|awk -F"\t" '{print $1}'`
        severity=`echo "$line"|awk -F"\t" '{print $2}'`
        server=`echo "$line"|awk -F"\t" '{print $5}'`
        message=`echo "$line"|awk -F"\t" '{print $4,$8}'`

        converter.pl -r MS_OVO_PROXY -c "$severity" -t T -i ORA_PERF_ALERT -h "$server" -e ORA -m "$messsage"

done < /opt/application.log

exit 0
2 
while read TIME SEV x MESS1 SERV x x MESS2 junk; do
  converter.pl -r MS_OVO_PROXY -c "$SEV" -t T -i ORA_PERF_ALERT -h "$SERV" -e ORA -m "$MESS1 $MESS2"
done < file
1 Like
3

May I suggest that the awk is massive overkill. If you are happy with a while read line loop, then try:-

while read timestamp severity x1 msg1 server x2 x3 msg2 rest
do
   message="$msg1 $msg2"
   converter.pl -r MS_OVO_PROXY -c "$severity" -t T -i ORA_PERF_ALERT -h "$server" -e ORA -m "$messsage"

done < /opt/application.log

Values x1, x2 & x3 are just placeholders. Variable rest captures everything from position 9 onwards (to leave msg2 as position 8)

I hope that this helps.

Robin
Liverpool/Blackburn
UK

1 Like
4

thanks for the suggestion, yeah thought so, is there an alternative using all using awk? i forgot to mention that the column exceeds $9 it reaches around 19 columns

i can get the column using below but i dont know how to pass it to converter.perl.

awk '{split($0,s,"\t");print "server:"s[5],"date:"s[1],"serverity:"s[2]}' /opt/application.log
5

Why would you want to use awk?

There's a number of ways. Such as:

awk '{print "converter.pl -r MS_OVO_PROXY -c \"" $2 "\" -t T -i ORA_PERF_ALERT -h  \"" $5 "\" -e ORA -m \""$4,$8 "\""}' file | sh

(without | sh for testing...)

$ cat file
A1 A2 A3 A4 A5 A6 A7 A8 J1
B1 B2 B3 B4 B5 B6 B7 B8 J2

$ awk '{print "converter.pl -r MS_OVO_PROXY -c \"" $2 "\" -t T -i ORA_PERF_ALERT -h  \"" $5 "\" -e ORA -m \""$4,$8 "\""}' file
converter.pl -r MS_OVO_PROXY -c "A2" -t T -i ORA_PERF_ALERT -h  "A5" -e ORA -m "A4 A8"
converter.pl -r MS_OVO_PROXY -c "B2" -t T -i ORA_PERF_ALERT -h  "B5" -e ORA -m "B4 B8"

But stick with the while-loop. It's cleaner, and just as efficient.

1 Like
6

looks like while loop is much efficient in this case. im actually learning awk so im figuring out how to do this via awk but i guess i should stick to while loop in this case then. thanks :wink:

---------- Post updated at 06:29 PM ---------- Previous update was at 06:20 PM ----------

The tab delimiter was ignored, it opted to use space delimiter instead via while loop

[oracle@server logs]$ while read timestamp severity XA MSG1 SERVER XB XC MSG2;do
>         message="$MSG1 $MSG2"
> echo "$timestamp"
> done < /opt/application.log
2013/08/07
2013/08/07
2013/08/07
2013/08/07
2013/08/07
2013/08/07
2013/08/07
2013/08/07
2013/08/07
2013/08/07
2013/08/07
2013/08/07
2013/08/07
[oracle@server logs]$
[oracle@server logs]$ awk -F"\t" '{print $1}' /opt/application.log
2013/08/07 04:48:36:685
2013/08/07 04:48:36:685
2013/08/07 04:53:36:514
2013/08/07 04:55:16:505
2013/08/07 05:07:36:505
2013/08/07 05:07:36:505
2013/08/07 05:07:36:505
2013/08/07 05:07:36:505
2013/08/07 05:07:36:505
2013/08/07 05:07:36:505
2013/08/07 05:07:36:505
2013/08/07 05:07:38:403
2013/08/07 05:07:38:642
[oracle@server logs]$
7

If your delimiters are tabs and spaces, you shouldn't need to specify it.

8

it doesnt work in my case. is this linux distro dependent? i am running redhat. i need to get the column tab delimited.

[oracle@server logs]$ awk '{print $1}' /opt/application.log
2013/08/07
2013/08/07
2013/08/07
2013/08/07
2013/08/07
2013/08/07
2013/08/07
2013/08/07
2013/08/07
2013/08/07
2013/08/07
2013/08/07
2013/08/07
[oracle@server logs]$

[oracle@server logs]$ awk -F"\t" '{print $1}' /opt/application.log
2013/08/07 04:48:36:685
2013/08/07 04:48:36:685
2013/08/07 04:53:36:514
2013/08/07 04:55:16:505
2013/08/07 05:07:36:505
2013/08/07 05:07:36:505
2013/08/07 05:07:36:505
2013/08/07 05:07:36:505
2013/08/07 05:07:36:505
2013/08/07 05:07:36:505
2013/08/07 05:07:36:505
2013/08/07 05:07:38:403
2013/08/07 05:07:38:642
[oracle@server logs]$
9

Oh, I see. You have date-time with a space, so you need tab. My mistake.

You can specify a field separator with the while loop too:

while IFS="\t" read ...; do
  ...
done < file

(if "\t" does not work, use a literal tab, or press Ctrl-V then Ctrl-I)

10

actually there are many spaces in the input file so i will need to work on using tabs. see 1 line sample below.

2013/08/07 05:07:38:642 CRITICAL        Oracle       Oracle:server.ecs.testpoc.com:Customer D: Oracle Database Server 11G - Basic:Database Counter: Instance Status        server.ecs.testpoc.com   Metric 'Alert has no defined Metric' changed status from 'good' to 'error'      Metric 'Alert has no defined Metric' crossed 'back to default' with value 'null'        DatabaseCounter Alert has no defined Metric     server.ecs.testpoc.com:287a96af-9af1-4431-b689-e1293f612786:Alert has no defined Metric::CRITICAL        server.ecs.testpoc.com:287a96af-9af1-4431-b689-e1293f612786:Alert has no defined Metric  CiHint:@@server.ecs.testpoc.com  ::null  Oracle@@server.ecs.testpoc.com                http://server.ecs.testpoc.com:8080/Oracle/servlet/Main?activeid=200708448&activerighttop=dashboard&view=new&dashboard_view=Details&dashboard_model=true&sis_silent_login_type=encrypted&login=%28sisp%29knjxbqDESkAn5mKcvgTmj%2FyFwHH5Ke3m%0D%0A&password=%28sisp%29EzqXbIXEFD%2BJbE1N1T%2FZlELjja0DKaN7%0D%0A        OracleMonitor:OracleDatabaseServer11GBasic:200708448 1

This actually breaks while loop limit which can accept 9 variables.

11

Not really. You could just use more parameters:

while read DATE TIME ...
12

nah, i cant do that the error messages will have variable spacing in them, especially for long or short error messages. so the only way is to delimit using tabs.

---------- Post updated at 07:03 PM ---------- Previous update was at 06:44 PM ----------

none of the above works, tried with IFS="\t", literal tab control v and control i. hmmm

13

Please post your code.

Also paste the output of:

$ head -1 file | awk 1 RS='\t'

(where file is the input file)

14

sure. here is the input file

[oracle@server logs]$ head -1 /opt/application.log | awk 1 RS='\t'
2013/08/07 04:48:36:685
NORMAL
Oracle
Oracle:server.testpoc.com:Health:Connection Statistics Monitor
server.testpoc.com
Metric 'SSH connections. Total failed to open V2' changed status from 'warning' to 'good'
Metric 'SSH connections. Total failed to open V2' crossed 'back to default' with value '0'
Connection Statistics Monitor
SSH connections. Total failed to open V2
server.testpoc.com::SSH connections. Total failed to open V2:Unknown:NORMAL
server.testpoc.com::SSH connections. Total failed to open V2
CiHint:@@server.testpoc.com
Unknown:Unknown:0
Oracle@@server.testpoc.com
Oracle:server.testpoc.com:SSH connections. Total failed to open V2
http://server.testpoc.com:8080/Oracle/servlet/Main?activeid=10006&activerighttop=dashboard&view=new&dashboard_view=Details&dashboard_model=true&sis_silent_login_type=encrypted&login=%28sisp%29knjxbqDESkAn5mKcvgTmj%2FyFwHH5Ke3m%0D%0A&password=%28sisp%29EzqXbIXEFD%2BJbE1N1T%2FZlELjja0DKaN7%0D%0A
OracleMonitor:__Health__:10006
1






[oracle@server logs]$

im echoing first column

while IFS="     " read line timestamp severity x msg1 server y z msg2
do
        message="$msg1 $msg2"
        echo "$timestamp"

done < /opt/application.log

exit 0

[oracle@server logs]$ ./Log_converter.sh
NORMAL
NORMAL
MINOR
MINOR
NORMAL
NORMAL
CRITICAL
NORMAL
NORMAL
NORMAL
NORMAL
CRITICAL
CRITICAL
NORMAL
CRITICAL
MINOR
NORMAL
MINOR
MINOR
CRITICAL
MINOR
CRITICAL
NORMAL
NORMAL
CRITICAL
NORMAL
NORMAL
NORMAL
NORMAL
CRITICAL
CRITICAL
[oracle@server logs]$
15

The timestamp is the first field.

while IFS="       " read timestamp severity x msg1 server y z msg2 junk; do
 ...
done

$ cat -n args.txt
     1	2013/08/07 04:48:36:685
     2	NORMAL
     3	Oracle
     4	Oracle:server.testpoc.com:Health:Connection Statistics Monitor
     5	server.testpoc.com
     6	Metric 'SSH connections. Total failed to open V2' changed status from 'warning' to 'good'
     7	Metric 'SSH connections. Total failed to open V2' crossed 'back to default' with value '0'
     8	Connection Statistics Monitor
     9	SSH connections. Total failed to open V2
    10	server.testpoc.com::SSH connections. Total failed to open V2:Unknown:NORMAL
    11	server.testpoc.com::SSH connections. Total failed to open V2
    12	CiHint:@@server.testpoc.com
    13	Unknown:Unknown:0
    14	Oracle@@server.testpoc.com
    15	Oracle:server.testpoc.com:SSH connections. Total failed to open V2
    16	http://server.testpoc.com:8080/Oracle/servlet/Main?activeid=10006&activerighttop=dashboard&view=new&dashboard_view=Details&dashboard_model=true&sis_silent_login_type=encrypted&login=%28sisp%29knjxbqDESkAn5mKcvgTmj%2FyFwHH5Ke3m%0D%0A&password=%28sisp%29EzqXbIXEFD%2BJbE1N1T%2FZlELjja0DKaN7%0D%0A
    17	OracleMonitor:__Health__:10006
    18	1
1 Like
16

oops didnt see that one. looking good now! :slight_smile: