Remove unwanted data?

Shell Programming and Scripting
1

Hi Can any one help me remove the unwanted data? I would want to remove the complete event id 4910 ( the type there is INFO), that means, I have to remove starting from 7th - 19th lines. can any one of you please help?

Thanks,

2

Try...

$ awk 'BEGIN{RS="Event Id"}!/Type : INFO/{if(p)printf RS p;p=$0}END{printf RS p}' file1 > file2
$ sdiff -w 60 file1 file2|nl
     1  Event Id: 4909                  Event Id: 4909
     2  Time : Tue May 8 23:06:06 20    Time : Tue May 8 23:06:06 20
     3  Type : RESET                    Type : RESET
     4  User : XXXXXX                   User : XXXXXX
     5  Message :                       Message :
     6  Auto-purged 23 log entries.     Auto-purged 23 log entries.
     7  Event Id: 4910               <
     8  Time : Tue May 8 23:08:13 20 <
     9  Type : INFO                  <
    10  User : XXXXXX                <
    11  Message :                    <
    12  From previous run            <
    13  Unable to open ODBC trace fi <
    14                               <
    15  DataStage Job 1 Phantom 2629 <
    16  rm: RT_SC1/jpfile: No such f <
    17                               <
    18  5 record(s) selected to SELE <
    19  DataStage Phantom Finished.  <
    20  Event Id: 4911                  Event Id: 4911
    21  Time : Tue May 8 23:08:13 20    Time : Tue May 8 23:08:13 20
    22  Type : STARTED                  Type : STARTED
$
3 
sed '7,19d' file > tmp
mv tmp file

id=4910
sed "/Event Id: $id/,/Event Id:/d" file > tmp
mv tmp file
4

I guess i posted it incorrectly. What i meant to say was i want remove the complete event where the type is INFO.

another example:

Expected output:

So all the events with type INFO is deleted. in the example the Events 4910 and 4912 has to be deleted

5 
sed "/^Event Id/s/.*/\\
&/" file | awk -v RS="" ' !/Type : INFO/ '
6

Well, its not removing the complete event what i mean is

Event Id: 4910
Time : Tue May 8 23:08:13 2007
Type : INFO
User : XXXXX
Message :
From previous run
Unable to open ODBC trace file XXXXX

DataStage Job 1 Phantom 26294
rm: RT_SC1/jpfile: No such file or directory

5 record(s) selected to SELECT list #1.
DataStage Phantom Finished.

7

Yes it not removing the complete event

8

You are right.

sed -e "/^ *$/d" -e "/^Event Id/s/.*/\\
&/" file | awk -v RS="" ' !/Type : INFO/ '
9

ANBU, thanks for the help. I am sorry, for the example it is working fine but not in the real scenario.

this is the exact values

10 
$ cat file
Event Id: 4909
Time : Tue May 8 23:06:06 2007
Type : RESET
User : XXXX
Message :
Auto-purged 23 log entries.
Event Id: 4910
Time : Tue May 8 23:08:13 2007
Type : INFO
User : XXXX
Message :
From previous run
Unable to open ODBC trace file ^O��/apps/Ascential/DataStage/branded_odbc/lib/odbctrac.so
DataStage Job 1 Phantom 26294
rm: RT_SC1/jpfile: No such file or directory
5 record(s) selected to SELECT list #1.
DataStage Phantom Finished.
Event Id: 4911
Time : Tue May 8 23:08:13 2007
Type : STARTED
User : XXXX
Message :
Starting Job vrc045005j.
$APT_DB2INSTANCE_HOME = /opt/usr/bin
$APT_CONFIG_FILE = /apps/Ascential/DataStage/Configurations/CONFIG.apt
DB_ALIAS = XXXX
DB_INST = XXXX
DB_NAME = XXXX
PASSWORD = XXXX
USER = XXXX
SERVER = XXXX
$ sed -e "/^ *$/d" -e "/^Event Id/s/.*/\\
> &/" file | awk -v RS="" ' !/Type : INFO/ '
Event Id: 4909
Time : Tue May 8 23:06:06 2007
Type : RESET
User : XXXX
Message :
Auto-purged 23 log entries.
Event Id: 4911
Time : Tue May 8 23:08:13 2007
Type : STARTED
User : XXXX
Message :
Starting Job vrc045005j.
$APT_DB2INSTANCE_HOME = /opt/usr/bin
$APT_CONFIG_FILE = /apps/Ascential/DataStage/Configurations/CONFIG.apt
DB_ALIAS = XXXX
DB_INST = XXXX
DB_NAME = XXXX
PASSWORD = XXXX
USER = XXXX
SERVER = XXXX
11

well, in the message section there are tabs at the first position which i am not able to display here.

12

if you have Python, here's an alternative:

#!/usr/bin/python
d={}
for line in open("file"):    
    if line.startswith("Event Id") and not d.has_key(line):
        d[line] = [line]
        temp = line
        continue
    d[temp].append(line)
for key in sorted(d.keys()):
    if not "Type : INFO\n" in d[key]:
        print ''.join(d[key])

output:

# ./test.py
Event Id: 4909
Time : Tue May 8 23:06:06 2007
Type : RESET
User : XXXXX
Message :
Auto-purged 23 log entries.

Event Id: 4911
Time : Tue May 8 23:08:13 2007
Type : STARTED
User : XXXXXXX
Message :
Starting Job XXXXXXX.
$APT_DB2INSTANCE_HOME = XXXXXX
$APT_CONFIG_FILE = XXXXXXXXXX
DB_ALIAS = XXXXXX
DB_INST = XXXXXXXX
DB_NAME = XXXXXXX
PASSWORD = XXXXXX
USER = XXXXXX
SERVER = XXXXXXXX

Event Id: 4918
Time : Tue May 8 23:08:16 2007
Type : WARNING
User : XXXXXXX
Message :
Sequential_File_1: When checking operator: When validating export schema: At field "XXXXXXXXX": Exporting nulla
ble field without null handling properties
13

I dont have python.

the initial tabs are removed when i try to copy paste in the forum. anyways, I have put the indent now(its tabs in the places of indents). THis gives more clear picture. is there a way i can attach file ????

14 
sed -n '
/Event Id:/ b block
H
$ b block
b
:block
x
/INFO/!p' infile
15

Thanks a lot - that worked like a charm!!!

16

have you tried Ygor's method?

17

The Ygor's method doesn't work on my AIX box.
My awk version uses only the first character of RS.

Try the following awk program :

#!/usr/bin/awk -f 
function print_lines() {
   for (i=1; i<= lines_count; i++)
      print lines;
}
/^Event Id:/ {
   if (! ignore_event)
      print_lines();
   lines_count  = 0;
   ignore_event = 0;
}
! ignore_event {
   if (/^Type : INFO/) 
      ignore_event = 1;
   else
      lines[++lines_count] = $0;
}
END {
   print_lines();
}

Jean-Pierre.

18

Yes i tried that but that doesnt work for me on my AIX.

Aigles, Could you please let me know how am i suppose to run your script??? I mean how do i pass the data to the awk script????? and also, radoulov's code???

19 
awk -f <awk_program_name> filename
20

That didnt work either. anyways, could any one of you please explain me the code of radoulov