awk Scripting

Hey guys,

I want to get all the columns in this input file tab-delimited, i need to get the column send them to a variable. From there i could print them in shuffle and pick and select columns i want.

Here is the input sample

2013/08/05 06:50:38:067 MINOR   SiteScope       SiteScope:domain.test.com:Customer B: Oracle Database Server 11G - Basic:Database Counter: Tablespace Monitoring  domain.test.com   Metric 'CATTBS1/% Free' changed status from 'good' to 'warning' Metric 'CATTBS1/% Free' crossed 'CATTBS1/% Free >= 90' with value '96.88'       DatabaseCounter CATTBS1/% Free  domain.test.com:9354ce4d-0298-44f0-92ff-26c5ccadd4db:CATTBS1/% Free:Unknown:MINOR domain.test.com:9354ce4d-0298-44f0-92ff-26c5ccadd4db:CATTBS1/% Free       CiHint:@@domain.test.com  Unknown:Unknown:96.88   SiteScope@@domain.test.comSiteScope:domain.test.com:CATTBS1/% Free  http://domain.test.com:8080/SiteScope/servlet/Main?activeid=200707457&activerighttop=dashboard&view=new&dashboard_view=Details&dashboard_model=true&sis_silent_login_type=encrypted&login=%28sisp%29knjxbqDESkAn5mKcvgTmj%2FyFwHH5Ke3m%0D%0A&password=%28sisp%29EzqXbIXEFD%2BJbE1N1T%2FZlELjja0DKaN7%0D%0A        SiteScopeMonitor:OracleDatabaseServer11GBasic.1:200707457       1
2013/08/05 06:50:38:067 MINOR   SiteScope       SiteScope:domain.test.com:Customer B: Oracle Database Server 11G - Basic:Database Counter: Tablespace Monitoring  domain.test.com   Metric 'CATTBS/% Free' changed status from 'good' to 'warning'  Metric 'CATTBS/% Free' crossed 'CATTBS/% Free >= 90' with value '96.88' DatabaseCounter CATTBS/% Free   domain.test.com:9354ce4d-0298-44f0-92ff-26c5ccadd4db:CATTBS/% Free:Unknown:MINOR  domain.test.com:9354ce4d-0298-44f0-92ff-26c5ccadd4db:CATTBS/% Free        CiHint:@@domain.test.com  Unknown:Unknown:96.88   SiteScope@@domain.test.com       SiteScope:domain.test.com:CATTBS/% Free    http://domain.test.com:8080/SiteScope/servlet/Main?activeid=200707457&activerighttop=dashboard&view=new&dashboard_view=Details&dashboard_model=true&sis_silent_login_type=encrypted&login=%28sisp%29knjxbqDESkAn5mKcvgTmj%2FyFwHH5Ke3m%0D%0A&password=%28sisp%29EzqXbIXEFD%2BJbE1N1T%2FZlELjja0DKaN7%0D%0A        SiteScopeMonitor:OracleDatabaseServer11GBasic.1:200707457       1
2013/08/05 06:50:38:067 CRITICAL        SiteScope       SiteScope:domain.test.com:Customer B: Oracle Database Server 11G - Basic:Database Counter: Tablespace Monitoring  domain.test.com   Metric 'CATTBS3/% Free' changed status from 'good' to 'error'   Metric 'CATTBS3/% Free' crossed 'CATTBS3/% Free >= 98' with value '99.61'       DatabaseCounter CATTBS3/% Free  domain.test.com:9354ce4d-0298-44f0-92ff-26c5ccadd4db:CATTBS3/% Free:Unknown:CRITICAL      domain.test.com:9354ce4d-0298-44f0-92ff-26c5ccadd4db:CATTBS3/% Free       CiHint:@@domain.test.com  Unknown:Unknown:99.61   SiteScope@@domain.test.com        SiteScope:domain.test.com:CATTBS3/% Free  http://domain.test.com:8080/SiteScope/servlet/Main?activeid=200707457&activerighttop=dashboard&view=new&dashboard_view=Details&dashboard_model=true&sis_silent_login_type=encrypted&login=%28sisp%29knjxbqDESkAn5mKcvgTmj%2FyFwHH5Ke3m%0D%0A&password=%28sisp%29EzqXbIXEFD%2BJbE1N1T%2FZlELjja0DKaN7%0D%0A        SiteScopeMonitor:OracleDatabaseServer11GBasic.1:200707457       1
2013/08/05 06:50:38:067 MINOR   SiteScope       SiteScope:domain.test.com:Customer B: Oracle Database Server 11G - Basic:Database Counter: Tablespace Monitoring  domain.test.com   Metric 'UNDOTBS1/% Free' changed status from 'good' to 'warning'        Metric 'UNDOTBS1/% Free' crossed 'UNDOTBS1/% Free >= 90' with value '96.78'     DatabaseCounter UNDOTBS1/% Free domain.test.com:9354ce4d-0298-44f0-92ff-26c5ccadd4db:UNDOTBS1/% Free:Unknown:MINOR        domain.test.com:9354ce4d-0298-44f0-92ff-26c5ccadd4db:UNDOTBS1/% Free      CiHint:@@domain.test.com  Unknown:Unknown:96.78   SiteScope@@domain.test.com        SiteScope:domain.test.com:UNDOTBS1/% Free http://domain.test.com:8080/SiteScope/servlet/Main?activeid=200707457&activerighttop=dashboard&view=new&dashboard_view=Details&dashboard_model=true&sis_silent_login_type=encrypted&login=%28sisp%29knjxbqDESkAn5mKcvgTmj%2FyFwHH5Ke3m%0D%0A&password=%28sisp%29EzqXbIXEFD%2BJbE1N1T%2FZlELjja0DKaN7%0D%0A        SiteScopeMonitor:OracleDatabaseServer11GBasic.1:200707457       1

again this is tab delimited and would output like this

[sitescope@server logs]$ awk -F"\t" '{print $1}' test.log
2013/08/05 06:50:38:067
2013/08/05 06:50:38:067
2013/08/05 06:50:38:067
2013/08/05 06:50:38:067
[sitescope@server logs]$

[sitescope@server logs]$ awk -F"\t" '{print $2}' test.log
MINOR
MINOR
CRITICAL
MINOR
[sitescope@server logs]$

[sitescope@server logs]$ awk -F"\t" '{print $4}' test.log
SiteScope:domain.test.com:Customer B: Oracle Database Server 11G - Basic:Database Counter: Tablespace Monitoring
SiteScope:domain.test.com:Customer B: Oracle Database Server 11G - Basic:Database Counter: Tablespace Monitoring
SiteScope:domain.test.com:Customer B: Oracle Database Server 11G - Basic:Database Counter: Tablespace Monitoring
SiteScope:domain.test.com:Customer B: Oracle Database Server 11G - Basic:Database Counter: Tablespace Monitoring
[sitescope@server logs]$

if you see a double space or more, it means a tab.

I can create a shell script with this setup like and print the variables.

date=`echo $line | awk -F"\t" '{print $1}`
severity=`echo $line | awk -F"\t" '{print $2}'`

But i believe this is really inefficient, i would like to make this all of this in awk good for me to expand my knowledge as well.

please point me to the right direction. thanks

You can use the split function in awk , then select columns to work with like s[1] s[8]

awk '{split($0,s,"\t");print s[1]}' test.log
2013/08/05 06:50:38:067
2013/08/05 06:50:38:067
2013/08/05 06:50:38:067
2013/08/05 06:50:38:067

thanks for this. would you mind sharing the awk structure as well.
i would like to have something like this written in awk.

while read line
do
	date=`echo $line | awk -F"\t" '{print $1}'`
	severity=`echo $line | awk -F"\t" '{print $2}'`
	server=`echo $line | awk -F"\t" '{print $5}'`
        echo "server:$server date:$date severity:$severity"
done < test.log

i have fairly enough experience when it comes to shell script but not in awk and i would like to see how all of this is done in awk. thanks

It may be that you can do everything inside awk and do not need to use variables.
You should also change from back tics `` to parentheses $()
date=$(echo $line | awk -F"\t" '{print $1}')

If you post your complete data and what you like to get out of it, we may help you more.

thanks. yes your right i might not need to assign it to any variables.
the

 date=$(echo $line | awk -F"\t" '{print $1}')

is really inefficient and what i am trying to avoid. you see if i have 50K lines it would execute awk 50K*no. of variables.

lets say i wanted to create in awk a format like this.

date being $1 severity is $2 and servername is $5, how would i accomplish this all in awk? i would prefer it stored in variables so i can easily control and print from there.

server:domain.test.com date:2013/08/05 06:50:38:067 severity:MINOR
server:domain.test.com date:2013/08/05 06:50:38:067 severity:MINOR
server:domain.test.com date:2013/08/05 06:50:38:067 severity:CRITICAL
server:domain.test.com date:2013/08/05 06:50:38:067 severity:MINOR

Is the goal to convert the log to another format?
Or do you do anything with the data?

You could use this and print the filed you like.

awk '{split($0,s,"\t");print "server:"s[4],"date:"s[1],"serverity:"s[2]}' test.log

Its some hard to test, since you example data does not contain tab as your original data does.

yes exactly, i need to convert the log to another format. the reason i did not paste the output here is because i dont know yet.
but what you have provided is working so i guess i can expand from there.

Why not

awk '{print "server:" $4 "\tdate:" $1 "\tseverity:"$2}' FS="\t" test.log

sep=`printf "\t"`
while IFS="$sep" read date severity field3 field4 lastfields
do
  echo "date=$date severity=$severity"
done < file

thanks for another suggestions