Problem with python script

Shell Programming and Scripting
1

I have a txt file of 8GB with same type of messages in it. for eg:

MTQ_BQUOTE, Length: 40, Timestamp: 4:00:52.064
MsgKey: symbol: XXX | reportingExchange: 11
Symbol: XXX, hash 004C5746
QS Symbol: XXX, market 1
Security Type: EQUAL (1)
Symbol Type: Equity.Share.Single.None
Session: XX_XXX (0)
Ticker Exchange: ABCD (11) => 11
Flags: x00000000
Bid: 2475, frac: 2, pure7 val: 24.75
Bid Size: 50
Ask: 0, frac: 0, pure7 val: ---
Ask Size: 0
Quote Condition: x00

There are tons of same messages with different values in the file. Can you please help me with a python script that can saperate the whole message with unique Flags value in an output file.

2

Does it have to be python? There are other suitable tools.

3

It does not necessarily be python. I use a tool called Cygwin (UNIX like). Can it be UNIX command?

4

can you explain more about " can saperate the whole message with unique Flags value in an output file."

5

I have a text file that has several messages, here I put 3 examples:

MTR_MKTSTAT, Length: 104, Timestamp: 3:30:00.000
MsgKey: statId: @ | type: T | basis: I | exchgId: O | modifier: T
Symbol: @XXX, hash 4F495415
QS Symbol: @XXXX, market 236
Security Type: MKTSTATS (8)
Symbol Type: Statistic.Share.Single.None
Session: US_Day (3)
Ticker Exchange: DTN (254) => 254
Flags: x00000000: 
Description: NASDAQ 3RD MKT ISSUES TICKS TOTAL 
Last: 446, frac: 0, pure7 val: 446
Open: 146, frac: 0, pure7 val: 146
High: 446, frac: 0, pure7 val: 446
Low: 146, frac: 0, pure7 val: 146
Close: 446, frac: 0, pure7 val: 446
Change: 0, frac: 0, pure7 val: --- 
Market Status: x01
Last Trade Time: 16:05:00.000
Last Trade Date: 10/4/2012
Open Time: 9:31:00.000
Close Time: 16:05:00.000
 
MTR_MKTSTAT, Length: 104, Timestamp: 3:30:00.000
MsgKey: statId: @ | type: T | basis: R | exchgId: O | modifier: T
Symbol: @TROT, hash 4F525415
QS Symbol: @TROT, market 236
Security Type: MKTSTATS (8)
Symbol Type: Statistic.Share.Single.None
Session: US_Day (3)
Ticker Exchange: DTN (254) => 254
Flags: x00000000: 
Description: NASDAQ 3RD MKT TICKS RATIO              
Last: 119, frac: 2, pure7 val: 1.19
Open: 106, frac: 2, pure7 val: 1.06
High: 124, frac: 2, pure7 val: 1.24
Low: 75, frac: 2, pure7 val: 0.75
Close: 119, frac: 2, pure7 val: 1.19
Change: 0, frac: 2, pure7 val:  --- 
Market Status: x01
Last Trade Time: 16:05:00.000
Last Trade Date: 10/4/2012
Open Time: 9:31:00.000
Close Time: 16:05:00.000
 
MTR_MKTSTAT, Length: 104, Timestamp: 3:30:01.000
MsgKey: statId: @ | type: I | basis: P | exchgId: T | modifier: T
Symbol: @XXX, hash 54504915
QS Symbol: @XXXX, market 236
Security Type: MKTSTATS (8)
Symbol Type: Statistic.Share.Single.None
Session: US_Day (3)
Ticker Exchange: DTN (254) => 254
Flags: x00000010: LAST 
Description: 
Last: 1396, frac: 0, pure7 val: 1396
Open: 5, frac: 0, pure7 val: 5
High: 1396, frac: 0, pure7 val: 1396
Low: 3, frac: 0, pure7 val: 3
Close: 0, frac: 0, pure7 val: --- 
Change: 1396, frac: 0, pure7 val: 1396
Market Status: x02
Last Trade Time: 3:30:01.000
Last Trade Date: 10/5/2012
Open Time: 0:00:00.000
Close Time: 23:59:59.000

If you look at the 9th line of each message, there is Flags with a code like x00000000. But the first 2 messages have the same code so, I want to saperate only those messages that have a unique code in a text file.

6

If you just want to filter out one type, and if messages are separated by at least one blank line:

awk 'index($9, "code_to_match")' RS=

Regards,
Alister

7

The below code creates the file with flagname (with the appropriate message block)

 
 $ nawk '/MTR/,/Close Time/{a[i++]=$0;if($0~/Flags:/){sub(":","",$2);fname=$2}if($0~/Close Time/){for(j=0;j<=i;j++)print a[j] >> fname;i=0}}' input.txt

In Solaris, use nawk , otherwise you can use awk

Above code will create two files ( x00000000, x00000010 ) - for the sample text you posted.

8

Thanks for your help itkamaraj. As you mentioned, above code generated 2 files with unique flags. however, my requirement is to get only one message containing unique flag value instead of multiple unique flags.

For example : After executing the code above, file ganerated with flag x00000000 contains 2 set of messages and 1 message set for x00000010, where as I just want to print any one message with flag x00000000 and x00000010 in one single output text file. Please share the modified code. Thanks

9

just replace the >> with >

10

Thanks for your reply, itkamaraj. After replacing >> with > also generates 2 different text files with all the messages of that unique flag code, as you mentioned above.
I am sorry if I have not mentioned it clearly but my requirement is to generate one text file with 1 message of each unique flag value (i don't care which message it picks if there are 1000 messages containing same unique code). I hope I have elaborated it enough for you to make a little correction in the code. Thanks again.