Simple awk

Hi, this must be a simple but this is my first interaction with shell and awk.

following is a log file needed to parse (2 lines separated by a line break for clarity):

2013-07-27 13:32:09,043 - ERROR - PerformanceUtility - Thread-14 - Performance - 9b348407-4f57-4983-a057-a55669821f68 | BidirectionalCompoundProviderMediation | ProviderAdapter.Response | B1.RPBLBKL04.EGS107 | RetryID: 1374949928859 -> 1374949928859 [ 0 ] | EngineRecordLoad: 1374949928860 -> 1374949928870 [ 10 ] | StateLoading: 1374949928870 -> 1374949928871 [ 1 ] | CorrelID: 1374949928859 -> 1374949928871 [ 12 ] | TotalID: 1374949928859 -> 1374949928871 [ 12 ] | com.bofa.rp.engine.v004.broker.optimus.task.OptimusResponseTask TaskExecution: 15 | com.bofa.rp.optimus.pattern.BidirectionalCompoundProviderMediation.ProviderAdapterResponseMap TaskExecution: 0 | Propogate: 1374949928887 -> 1374949928945 [ 58 ] | Delivery: 1374949928887 -> 1374949928945 [ 58 ] | Orchestration: 1374949928871 -> 1374949928945 [ 74 ] | UpdateEngineRecord: 1374949928945 -> 1374949928948 [ 3 ] | Propogate: 1374949928950 -> 1374949929041 [ 91 ] | PublishAudit: 1374949928948 -> 1374949929042 [ 94 ] | Propogate: 1374949929043 -> 1374949929043 [ 0 ] | PublishRecovery: 1374949929042 -> 1374949929043 [ 1 ] | Execution: 1374949928859 -> 1374949929043 [ 184 ]

2013-07-27 13:32:08,723 - ERROR - PerformanceUtility - Thread-13 - Performance - b089c04a-a936-45f8-a94b-196da5494424 | BidirectionalCompoundProviderMediation | ProviderAdapter.Request | B1.RPBLBKL04.EGS105 | RetryID: 1374949928412 -> 1374949928412 [ 0 ] | EngineRecordLoad: 1374949928412 -> 1374949928419 [ 7 ] | StateLoading: 1374949928419 -> 1374949928419 [ 0 ] | CorrelID: 1374949928412 -> 1374949928419 [ 7 ] | TotalID: 1374949928412 -> 1374949928419 [ 7 ] | com.bofa.rp.engine.v004.broker.optimus.task.OptimusResponseTask TaskExecution: 14 | com.bofa.rp.optimus.pattern.BidirectionalCompoundProviderMediation.ProviderAdapterResponseMap TaskExecution: 0 | Propogate: 1374949928434 -> 1374949928612 [ 178 ] | Delivery: 1374949928433 -> 1374949928612 [ 179 ] | Orchestration: 1374949928419 -> 1374949928612 [ 193 ] | UpdateEngineRecord: 1374949928612 -> 1374949928616 [ 4 ] | Propogate: 1374949928618 -> 1374949928620 [ 2 ] | PublishAudit: 1374949928616 -> 1374949928620 [ 4 ] | Propogate: 1374949928620 -> 1374949928723 [ 103 ] | PublishRecovery: 1374949928620 -> 1374949928723 [ 103 ] | Execution: 1374949928412 -> 1374949928723 [ 311 ]

This is my attempt ...

#!/bin/awk
FILENAME="EnginePerformance.log"
cat $FILENAME | grep -E "(RPALBKL|RPBLBKL|RPCLBKL)" | awk -F '|'
'{ 
       if ($3 ~ /.*Request.*/)
          print $3 $4 $11 $12 $22 $23 $24 $25 $26 $27         
       else if ($3 ~ /Response/) {
          print $3 $4 $6 $19 $20 $21 $22 $23 
       }
}'

Thanks for your help!

Hi liv2luv,

check this out:

#!/bin/bash

or

 #!/bin/ksh

depending on the type of sheel ,you should keep at the begining. and not #!/bin/awk

$ awk -F '|' '/RPALBKL/||/RPBLBKL/||/RPCLBKL/{if($3~/\.Request/)print $3 $4 $11 $12 $22 $23 $24 $25 $26 $27;else if($3~/\.Response/)print $3 $4 $6 $19 $20 $21 $22 $23 } ' file
 ProviderAdapter.Response  B1.RPBLBKL04.EGS107  EngineRecordLoad: 1374949928860 -> 1374949928870 [ 10 ]  PublishRecovery: 1374949929042 -> 1374949929043 [ 1 ]  Execution: 1374949928859 -> 1374949929043 [ 184 ] 2013-07-27 13:32:08,723 - ERROR - PerformanceUtility - Thread-13 - Performance - b089c04a-a936-45f8-a94b-196da5494424  BidirectionalCompoundProviderMediation  ProviderAdapter.Request  B1.RPBLBKL04.EGS105
 

---------- Post updated at 02:38 AM ---------- Previous update was at 02:35 AM ----------

Adding to that , you can get similar type of output : by adding "|"with the print statement as your output is pipe delimited:

print $3,"|", $4 ,"|", $11,"|", ....

Many thanks, the code worked just perfect!

Hopefully I will be able to learn this well one day..

Here is a very slightly more complex awk script that should be much more efficient than what you were doing.

#!/bin/ksh
FILENAME=EnginePerformance.log
awk 'BEGIN { FS = " [|] "; OFS = " | " }
! /RPALBKL|RPBLBKL|RPCLBKL/ { next }
$3 ~ /Request/  { print $3,$4,$11,$12,$22,$23,$24,$25,$26,$27 }
$3 ~ /Response/ { print $3,$4,$6,$19,$20,$21,$22,$23 }' "$FILENAME"

I used ksh, but this script will work with any shell that recognizes basic Bourne shell syntax. If you are using a Solaris/SunOS system, use /usr/xpg4/bin/awk , /usr/xpg6/bin/awk , or nawk instead of just awk . (This is also true if you use the awk scripts suggested by rveri.)

The first line in the awk script above sets the input and output field separators to " | " . The second line replaces your grep. And the third and fourth lines replace your if statements.
Note that your sample input only had 20 fields, but you're printing the contents of fields up to field #27. That is why there are empty fields at the end of the output lines this script produces.

Note also that your sample input had lines that were about 1150 bytes long. If your real input has lines that are longer than LINE_MAX on your system (you can get the limit on your system using the command getconf LINE_MAX ), the behavior of awk and grep is unspecified. The standards specify LINE_MAX must be at least 2048; and 2048 is the limit on many systems. If you can't use awk and grep to get reliable results on your system, the solution to your problem will be much more complex.

You didn't show us what output you wanted, so rveri and I made similar guesses at what you wanted. The difference between rveri's scripts and mine is that I assume your field separator (for input and output) was <space><vertical bar><space> where rveri removed the <vertical bar> in the first suggestion and added more spaces in the output with the suggested edit.

With the sample input given in the 1st message in this thread, the output my suggestion above produces is:

ProviderAdapter.Response | B1.RPBLBKL04.EGS107 | EngineRecordLoad: 1374949928860 -> 1374949928870 [ 10 ] | PublishRecovery: 1374949929042 -> 1374949929043 [ 1 ] | Execution: 1374949928859 -> 1374949929043 [ 184 ] |  |  | 
ProviderAdapter.Request | B1.RPBLBKL04.EGS105 | com.bofa.rp.optimus.pattern.BidirectionalCompoundProviderMediation.ProviderAdapterResponseMap TaskExecution: 0 | Propogate: 1374949928434 -> 1374949928612 [ 178 ] |  |  |  |  |  | 

The original code posted by rveri produces:

 ProviderAdapter.Response  B1.RPBLBKL04.EGS107  EngineRecordLoad: 1374949928860 -> 1374949928870 [ 10 ]  PublishRecovery: 1374949929042 -> 1374949929043 [ 1 ]  Execution: 1374949928859 -> 1374949929043 [ 184 ]
 ProviderAdapter.Request  B1.RPBLBKL04.EGS105  com.bofa.rp.optimus.pattern.BidirectionalCompoundProviderMediation.ProviderAdapterResponseMap TaskExecution: 0  Propogate: 1374949928434 -> 1374949928612 [ 178 ] 

Note the leading spaces and the lack of any indication at the end of the output lines corresponding to the empty output fields.

With the suggested edit to rveri's code, it produces:

 ProviderAdapter.Response  |  B1.RPBLBKL04.EGS107  |  EngineRecordLoad: 1374949928860 -> 1374949928870 [ 10 ]  |  PublishRecovery: 1374949929042 -> 1374949929043 [ 1 ]  |  Execution: 1374949928859 -> 1374949929043 [ 184 ] |  |  | 
 ProviderAdapter.Request  |  B1.RPBLBKL04.EGS105  |  com.bofa.rp.optimus.pattern.BidirectionalCompoundProviderMediation.ProviderAdapterResponseMap TaskExecution: 0  |  Propogate: 1374949928434 -> 1374949928612 [ 178 ]  |  |  |  |  |  | 

Note the leading spaces and the added spaces separating existing input fields that are copied to the output. This presents the empty fields at the end of the line the same way my suggestion does it.

Maybe one of these three will be close to what you want.

Thanks Don, honestly, I can not appreciate the improvement in code given my limitation in understanding it.

The code given by rveri which produces space separated values (or rather just the tokens with out pipe) does suffice. Thanks again for your help!

---------- Post updated at 05:04 PM ---------- Previous update was at 11:22 AM ----------

The servers are cloned from a single image, however on some of the servers, I am gettign this:

./show-log.sh[8]: : cannot execute [Is a directory]

What could be the reason. Shell is ksh. Google search did not return any satisfactory responses.

Seeing the contents of show-log.sh would be helpful in deducing why show-log.sh is misbehaving.

$ cat show-log.sh
#!/bin/ksh
## This script is to view the wanted values in EnginePerformance Log for both Request and Response logs from message broker.
FILENAME="EnginePerformance.log"

## Check if the file exist
if [ -e $FILENAME ];
  then ""
else
  echo "$FILENAME does not exist!"
  exit 1
fi

## Tail the file and parse request and response logs
tail -1000f $FILENAME | awk -F '|' '/RPALBKL/||/RPBLBKL/||/RPCLBKL/{
if($3~/\.Request/)
   print $3 $4 $11 $12 $22 $23 $24 $25 $26 $27;
else if($3~/\.Response/)
   print $3 $4 $6 $19 $20 $21 $22 $23 }
'
exit 0

The code is same across all servers. Also the servers are cloned from a single image. It is indeed surprising to see the in-consistent behavior. Any hint will be helpful. Thanks.

What were you expecting the command:

""

to do on line 7 in your script ( then "" )?

I have resorted to that odd looking

then ""

because I could not get

if [ ! -e $FILENAME ];

this statement working...

In case you haven't noticed, this isn't working either. That is what is producing the diagnostic:

./show-log.sh[8]: : cannot execute [Is a directory]

Please show us the exact output you get from running the following script:

#!/bin/ksh
echo "$PWD"
uname -a
ls -l /bin/ksh EnginePerformance.log
## This script is to view the wanted values in EnginePerformance Log for both Request and Response logs from message broker.
FILENAME="EnginePerformance.log"

## Check if the file exist
if [ ! -e "$FILENAME" ]
then
        echo "$FILENAME does not exist!"
        exit 1
fi
echo "$FILENAME exists."

Following is the output of the above code:

/usr/bfa/ltb/rqpt/orange/logs
Linux lkcmsxmbltb4.uskcmtb.amrs.xyz.com 2.6.32-358.11.1.el6.bz976915.bz956979.1.x86_64 #1 SMP Wed Jul 3 14:56:47 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux
lrwxrwxrwx 1 root root     21 Feb 11 09:58 /bin/ksh -> /etc/alternatives/ksh
-rw-rw---- 1 mqsi unixuser  0 Jul 30 11:23 EnginePerformance.log
EnginePerformance.log exists.

It's working -- it does exist. It's empty, but that's not what you asked the shell for...

If you want to check if it's empty too, -s is true on nonempty files:

if [ -e "$FILENAME" ] && [ -s "$FILENAME" ]
then
        echo "File exists and has contents"
else
        echo "File is empty or missing"
fi

OK. So it looks like if [ ! -e filename ] is working as expected. What made you think it wasn't working?