How to use awk with multiple pattern?

Shell Programming and Scripting
1

Hi All,

need your help , i want grouping complicated log below became count data message error per 5 minutes.

log some like below :

/data/logs/ag/tdr25001.log:2019-07-20 19:49:25,077|57|twallet|{"originatorConversationID":"STRANGE_CHECK_PIN_1563626965017","msisdn":"62859130860152","commandID":"InitTrans_ApplyTokenWithoutNotification"}|{"data":{"res:Body":{"res:ResultCode":-1,"res:ReferenceData":"","res:ResultType":0,"res:ResultDesc":"Mohon maaf, transaksi anda saat ini belum dapat diproses"},"res:Header":{"res:OriginatorConversationID":"STRANGE_CHECK_PIN_1563626965017","res:Version":1,"res:ConversationID":"AG_20190720_000076559a3563bd26fe"}},"message":"Mohon maaf, transaksi anda saat ini belum dapat diproses","status":"-1"}
/data/logs/ag/tdr25001.log:2019-07-20 00:47:17,227|50|twallet|{"uid":"100024215547","remark":"LinkNFCTag","originatorConversationID":"STRANGE_LinkNFCTag_1563558437174","msisdn":"6282226261
100","commandID":"LinkNFCTagSticker"}|{"data":{"res:Body":{"res:ResultCode":-1,"res:ResultType":0,"res:ResultDesc":"System internal error."},"res:Header":{"res:OriginatorConversationID":"STRANGE_LinkNFCTag_1563558437174","res:Version":1,"res:ConversationID":"ag11907200047176110076184"}},"message":"System internal error.","status":"-1"}
/data/logs/ag/tdr25001.log:2019-07-20 00:15:47,343 {"data":{"res:Body":{"res:ResultCode":-1,"res:ResultType":0,"res:ResultDesc":"System internal error."},"res:Header":{"res:OriginatorConve
rsationID":"STR0191062831239189741563556547218","res:Version":1,"res:ConversationID":"ag11907200015471897471631"}},"message":"System internal error.","status":"-1"}
/data/logs/ag/tdr25001.log:2019-07-20 00:15:48,381 {"data":{"res:Body":{"res:ResultCode":-1,"res:ResultType":0,"res:ResultDesc":"System internal error."},"res:Header":{"res:OriginatorConve
rsationID":"STR0226062831239189741563556548192","res:Version":1,"res:ConversationID":"ag11907200015481897471634"}},"message":"System internal error.","status":"-1"}
/data/logs/ag/tdr25001.log:2019-07-20 00:47:17,227 {"data":{"res:Body":{"res:ResultCode":-1,"res:ResultType":0,"res:ResultDesc":"System internal error."},"res:Header":{"res:OriginatorConve
rsationID":"STRANGE_LinkNFCTag_1563558437174","res:Version":1,"res:ConversationID":"ag11907200047176110076184"}},"message":"System internal error.","status":"-1"}
/data/logs/ag/tdr25001.log:2019-07-20 01:00:26,394 {"data":{"res:Body":{"res:ResultCode":-1,"res:ReferenceData":{"res:ReferenceItem":[{"com:Key":"bill_ref","com:Value":"0031005931706050004
1907A              92290 AGUNG DYATMIKA EKA NUGRAHA"},{"com:Key":"amount","com:Value":"000000092290"},{"com:Key":"Requester","com:Value":628110000009},{"com:Key":"OriginatorConversationID"
,"com:Value":"ASSTR0777I06281216666691563559209619"},{"com:Key":"TransactionDateAndTime","com:Value":"0720010009"},{"com:Key":"BillReferenceNumber","com:Value":"0315931706"},{"com:Key":"Sy
stemTraceAuditNumber","com:Value":792566}]},"res:ResultType":0,"res:ResultDesc":"Mohon maaf, transaksi anda saat ini belum dapat diproses"},"res:Header"
:{"res:OriginatorConversationID":"ASSTR0777I06281216666691563559209619","res:Version":1,"res:ConversationID":"ag11907200100256666977667"}},"message":"Mohon maaf, transaksi anda saat ini belum dapat diproses","status":"-1"}

i already test some awk below but still confuse how to get second parameter / value result

grep "\"res:ResultCode\":-1" /data/logs/ag/tdr2500*.log | awk -F '[|]' '{print $1,$5}'

my expectation :

2019-07-20 04:29 "Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi"  5
2019-07-20 04:29 "System internal error."  10
2019-07-20 04:34 "Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi"  1

2019-07-20 04:34 "System internal error."  5

need help...

2

Again, your sample file doesn't match your expectation. Try

awk '
/"res:ResultCode":-1/   {sub (/:[0-9]*,[0-9]*/, " ", $0)
                         match ($0, /"message":"[^"]*/)
                         CNT[$1 " " $2 " " substr($0, RSTART+11, RLENGTH-11)]++
                        }
END                     {for (c in CNT) print c, CNT[c]
                        }
' file
2019-07-20 00:47 System internal error. 2
2019-07-20 00:15 System internal error. 2
2019-07-20 01:00 Mohon maaf, transaksi anda saat ini belum dapat diproses 1
2019-07-20 19:49 Mohon maaf, transaksi anda saat ini belum dapat diproses 1
1 Like
3

Hi rudy,

thanks for your reply , your awk great.

btw if the output result want to add delimiter "|" how to add for the awk, i try below but dont know how to add delimitter for column 3 and 4 .

2019-07-21 08:16 Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi 1
2019-07-21 08:18 Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi 1
2019-07-21 08:18 System internal error. 1
2019-07-21 08:19 System internal error. 2
2019-07-21 08:20 Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi 1
2019-07-21 08:21 System internal error. 1

try with :

cat files | awk '/"res:ResultCode":-1/   {sub (/:[0-9]*,[0-9]*/, " ", $0) match ($0, /"message":"[^"]*/) CNT[$1 "|" $2 "|" substr($0, RSTART+11, RLENGTH-11)]++}END{for (c in CNT) print c, CNT[c]}' | sort -nk1

the result just missed column 3 and 4 :

2019-07-21|08:16|Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi 1
2019-07-21|08:18|Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi 1
2019-07-21|08:18|System internal error. 1
2019-07-21|08:19|System internal error. 2
2019-07-21|08:20|Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi 1
2019-07-21|08:21|System internal error. 1

expected result :

2019-07-21|08:16|Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi|1
2019-07-21|08:18|Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi|1
2019-07-21|08:18|System internal error.|1
2019-07-21|08:19|System internal error.|2
2019-07-21|08:20|Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi|1
2019-07-21|08:21|System internal error.|1

many thanks for your effort

Regards

--- Post updated at 10:10 PM ---

once question again , and how to check some threshold when have more than 1 row, sample :

have 3 row, and count more than 300 ( threshold = 300 ), if > 300 will be sent alert.

2019-07-21|08:37|Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi|500
2019-07-21|08:37|System internal error.|1
2019-07-21|08:37|belum dapat diproses.|1

maybe if only 1 row i can use this awk to check files :

data=`cat data.txt | awk -F "|" '{print $4}'`
if [ $data -gt 300 ]
then
echo "sent alert"

but don't know how to check with 3 row...please kind to help

Regards

4

Try

awk '
/"res:ResultCode":-1/   {sub (/:[0-9]*,[0-9]*/, " ", $0)
                         match ($0, /"message":"[^"]*/)
                         CNT[$1 OFS $2 OFS substr($0, RSTART+11, RLENGTH-11)]++
                        }
END                     {for (c in CNT) {print c, CNT[c]
                                         if (CNT[c] > 300) {print "alert " c | ("mail user@domain.com") }
                                        }
                        }
' OFS="|" file
1 Like
5

Hi RudyC,

thanks a lot for your reply.

your awk running well...but how print count and sent alert with also "messages error" ( row 3 )

2019-07-20|21:17|Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi|145
2019-07-20|21:17|System internal error|6
2019-07-20|21:17|Silahkan ulangi beberapa saat lagi|3

expectation alert

please check have many error "Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi" count 145

Thanks alot

Regards

6

Hi rudyC,

i found some awk almost done for this script but dont know how to print result $0 to alert ( i am using curl )

awk it will like below :

cat datafiles  | strings | awk '/"res:ResultCode":-1/   {sub (/:[0-9]*,[0-9]*/, " ", $0) match ($0, /"message":"[^"]*/) CNT[$1 OFS $2 OFS substr($0, RSTART+11, RLENGTH-11)]++}END{for (c in CNT) print c, CNT[c]}' OFS="|" | sort -nk1 | awk -F"|" '{if ($4 > 100)print ""$1","$2",please check have many error respond MFS",$3,"count :",$4;'}''

result :

2019-07-20,21:06,please check have many error respond MFS Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi count : 150
2019-07-20,21:07,please check have many error respond MFS Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi count : 151
2019-07-20,21:08,please check have many error respond MFS Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi count : 150

but how to sent alert for this result print $0 if using curl

expectation alert :

curl -X GET "http://x.x.x.x:10000/telegram/submit_fajar.php?msg=print $0+`hostname`+time_`date +%d%h%y_%H.%M.%S`"

noted :

print $0 mean result count > 100.

Thanks
Fajar

7

I'm afraid I don't understand your requirements.

You don't need the second awk invocation. Make the first's END section

if (CNT[c] > 300) {print "Please check have many error " c " count " CNT[c] | ("mail ...") }
8

Hi RudiC

thanks for reply

my requirement if found condition $4 > 300 will sent alert with curl

sample :

2019-07-20|20:25|Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi|483

2019-07-20|20:25|Internal Error|8

it will sent curl with this condition

curl -X GET "http://x.x.x.x:10000/submit_fajar.php?msg=print $0+`hostname`+time_`date +%d%h%y_%H.%M.%S`"

noted :

print $0 same as "2019-07-20|20:25|Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi|483"

Thanks

9

You could try:

awk '
/"res:ResultCode":-1/   { 
   sub (/:[0-9]*,[0-9]*/, " ", $0)
   match ($0, /"message":"[^"]*/)
   CNT[$1 OFS $2 OFS substr($0, RSTART+11, RLENGTH-11)]++
}
END {
  for (c in CNT) {
     print c, CNT[c]
     if (CNT[c] > 300)
        system( \
           "curl -X GET \"http://x.x.x.x:10000/submit_fajar.php?msg=" \
           c OFS CNT[c] "+`hostname`+time_`date +%d%h%y_%H.%M.%S`\"")
  }
}
' OFS="|" infile