Need to Grep or awk a logfile for greater than value

Shell Programming and Scripting
1

Hello all

Hoping someone would be kind enough to suggest a solution to a problem i have, and see if maybe i can even do this without a script.

Essentially i have a very large log file, and within it each line had a value called TTMT, and it records a variable number in the following way, so if a message took 20 mseconds it would appear as TTMT {20} or 21 mseconds then TTMT {21}

There are thousands of entries of TTMT in this logfile and i am only concerned with timings above 20 mseconds, so would like to look only at message strings where the number is geater than 20, and append/write them to a seperate file.

I can do a grep 'TTMT {21}' friday.log |more to see each message where its 21 but i would like a way to see anything above 20 and snend to another log > above20.log

I believe i could use awk but can not see a way to do this correctly, is it possible to do the above using awk or/and grep? rather than having to create a script?

Any advice gratefully accepted :slight_smile:

2

It would help if you provide a sample input content. If the TTMT value is at a fixed column, that would be trivial with awk.

1 Like
3

Hi, thanks for answer, so each message in the log is written like this, it has a timestamp and then each variable is within a { } Not sure if it has a certain fixed column though? is it a case of each { } = a column so i need to count them to see where TTMT comes? If so i counted and TTMT appears to be the 55th {}

Here is a example line, where it takes TTMT {21} - would like to see any entries like this where TTMT exceeds 20 (majority are below this number)

16:00:01:350528|0890-ATQU OrdClose Code {1 0003844534} OrdId {2} MsgCode {0} Msg {OK} R {0} T {3551349298} Act {chg} Code {10003844534} OrdId {2} T {35519298} S {on} Ed {1} Owner {CRET} Leg {doubl
e} ByRec {1} ByMid {0} ByBest {0} Mid {0.010000} IS {100.000000 100.000000} OS {100 100} Qshow {CRETA} QBid {102.410000 10.000 10.000} QAsk {102.650000 10.000 10.000} BBid {102.430000 15.000} BAsk {102.6300
00 17.500} MTrader {CREDM} MBid {102.410000 10.000 10.000} MAsk {102.640000 10.000 10.000} DelOnEdit {0} BySkew {0} ByFastSpread {0} Skew {0.000000} FastSpread {0.000000} StatusDetailLong {} ByBestType {1} Ch
eckJump {0} VOrdId {2} Fly {0} LinkId {} MLinkId {} ThClass {default} BackTrader {CRED} Refreshing {0} Component {} PriceComponent {} BestToCheck {0} BestTime {0} DiffTime {0} SubscrStatus {2} Attributes {0
0} TT {52} TTMKT {52} TTQ {0} TTSendQ {0} TTMT {21} TTNET {33} TTRecvQ {0}

4

Assuming each line format is the same, you might use this:

awk '{ttmt=$125; gsub("[{}]","",ttmt); if(ttmt>20) printf("%s: %s\n",ttmt,$0);}' filename

PS: use nawk instead of awk if you happen to run Solaris.

1 Like
5

Hey that is going to work i think, but its actually referencing TTsendQ which is the field before TTMT , how do i move that to TTMT? is it by altering ttmt=$125 to 127 for example?

I changed ttmt to ttmt=$127 and it worked perfect, is just showing the few ttmt above 20 now, many thanks jlliagre, your a star!

P.S using Suse Linux rather than Solaris, but this is more than enough, thanks again

6 
grep -E 'TTMT {2[1-9]|TTMT {[3-9][0-9]|TTMT {[1-9][0-9][0-9]' input >above20.txt

[ctsgnb@shell ~/sand]$ cat input
whatever TTMT {20} whatever
whatever TTMT {20} whatever
whatever TTMT {25} whatever
whatever TTMT {20} whatever
whatever TTMT {23} whatever
whatever TTMT {20} whatever
whatever TTMT {21} whatever
whatever TTMT {24} whatever
whatever TTMT {25} whatever
whatever TTMT {26} whatever
whatever TTMT {20} whatever
whatever TTMT {30} whatever
whatever TTMT {12000} whatever
whatever TTMT {20} whatever
whatever TTMT {200} whatever
[ctsgnb@shell ~/sand]$ grep -E 'TTMT {2[1-9]|TTMT {[3-9][0-9]|TTMT {[1-9][0-9][0-9]' input
whatever TTMT {25} whatever
whatever TTMT {23} whatever
whatever TTMT {21} whatever
whatever TTMT {24} whatever
whatever TTMT {25} whatever
whatever TTMT {26} whatever
whatever TTMT {30} whatever
whatever TTMT {12000} whatever
whatever TTMT {200} whatever
[ctsgnb@shell ~/sand]$
7

You do not have to hard code the position. Let AWK scan through all the fields

awk '
{
        for ( i=1 ; i<=NF ; ++i ) {
                if ( $i == "TTMT" ) {
                        j=i+1
                        gsub("[{}]", "", $j)
                        if ( $j > 20 ) {
                                print $i, $j
                        }
                }
        }
}' inputfile