Selecting Lines on text file

Hi All,

I am creating a script that sends log data from text files to a Database and I will like to read sugestions, as I think that there might be better ways to achive this than with my shell script; maybe perl or I don't know, but I will like to read some sugestions.

The log is from Nagios, here is an example of the data I am interested in:

[1230791427] EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;SACNT141;SQL Event Log;2;Application [error] [mssqlserver #17055]: 3041 : BACKUP failed to complete the command BACKUP DATABASE [Apple_InterchangeSQ] TO [Apple_InterchangeSQ_Full] WITH  INIT
[1230791545] HOST ALERT: FSERVWEB;DOWN;SOFT;1;10.141.8.40 is unreachable (loss: 100%)
[1230791685] SERVICE DOWNTIME ALERT: ZCREH000;SYSTEM-DISK;STOPPED; Service has exited from a period of scheduled downtime
[1230793487] HOST DOWNTIME ALERT: MILSTARS;STOPPED; Host has exited from a period of scheduled downtime
[1230793964] SERVICE DOWNTIME ALERT: TXSLRCAD1;DISK-USAGE-E-90-95;STOPPED; Service has exited from a period of scheduled downtime

As you can see, there is a Timestamp, the type of alert and the information separated by commas. The diferent log entries have different Columns, thus I separate them according to the alert type, and then arrange the data to fit into my Database.

What will be the best way to read line by line, and according to the Alert type; create a CSV line with the info to be sent to the database (or put it directly in case of perl or PHP)....

Any advise is welcome, as I am currently using a shell script with grep and awk... and I feel is not the best way to do it.

Regards,
oconmx

i hope help you:

while read line
do
        TypeAlert=`echo "$line" | awk -F":" '{ split ( $1,vect,"]");   print vect[2] }'|sed "s/^  *//"`
        if [ "$TypeAlert" = "EXTERNAL COMMAND" ]
        then
                echo "EXTERNAL COMMAND ALERT"
                echo $line >> file1
        else
         if [ "$TypeAlert" = "HOST ALERT" ]
         then
                echo "HOST ALERT"
                echo $line >> file2
         else
           if [ "$TypeAlert" = "SERVICE DOWNTIME ALERT" ]
           then
                echo "SERVICE DOWNTIME ALERT"
                echo $line >> file3
           else
              if [ "$TypeAlert" = "HOST DOWNTIME ALERT" ]
              then
                 echo "HOST DOWNTIME ALERT"
                echo $line >> file4
              else
                if [ "$TypeAlert" = "SERVICE DOWNTIME ALERT" ]
                then
                        echo "[SERVICE DOWNTIME ALERT]"
                        echo $line >> file5
                fi
              fi
          fi
         fi
        fi
done<file

Let me try it...

Looks good and way cleaner than my script

Thank you!

We can use only awk:

awk -F'\]|:' '{line=$0;sub("^ ","",$2);sub(" ","_",$2);file=$2;print line >> file;close(file)}' file

danmero,

I dont understand all your code, I cant imagine how it separates the alert types... besides that, it is giving me an error:

awk: (FILENAME=file FNR=196) fatal: expression for `>>' redirection has null string value

Thank you!
Oconmx

Can you post the source filename and the line 196

awk 'NR==196' file

Hi danmero,

It was an empty line, the file has many... I already fixed a few and crop the file to try your script, it's good, I like the Idea. I will try to add some modifications and try the shell script with a CASE statement to see which is faster...

Is there anyway to make awk skip blank lines?

Thank you!
Carlos

Yep

awk -F'\]|:' 'NF{line=$0;sub("^ ","",$2);sub(" ","_",$2);file=$2".out";print line >> file;close(file)}'  file

Excelent !

Thank you again danmero!

---------- Post updated at 10:51 AM ---------- Previous update was at 09:36 AM ----------

Hello Again,

I am using this line:
awk -F'\]|:' 'NF{line=$0;sub("^ ","",$2);sub(" ","_",$2); file="awk/"$2".out"; if ($2 !~ /W|Su|Nag|Fin|Ca|CU|Au|LOG|Loc|G/); print line >> file; close(file); }' file

to remove unwanted types...

now, on my previous script I was converting the [Timestamp] to date and adding ";" as separator piping awk...:

awk '{FS=";"}{gsub(/SERVICE NOTIFICATION: |HOST NOTIFICATION: /,"")}{gsub(/\[1/,"1")}{gsub(/] /,";")}{t=strftime("%m/%d/%Y %T", $1 )}{print t";GMT -7;"w$0}'|awk 'BEGIN {FS=OFS=";"} $6=$7~/^h|^n/?"HOST STATUS;"$6:$6'|awk '{FS=OFS=","}{gsub(/;/,",")}{print $1","$2","$3","$4","$5","$6","$7","$8",\""$9 $10 $11 $12 $13 $14 $15 $16 $17"\""}'

Is there any easy way to remove the [] from the timestamp, convert it to time and change the , to ; .... as you can see I am not that good at awk; so any advise is highly appreciated, to avoid piping commands.

Regards,
Oconmx

To keep the forums high quality for all users, please take the time to format your posts correctly.

1. Use Code Tags when you post any code or data samples so others can easily read your code.
   You can easily do this by highlighting your code and then clicking on the # in the editing menu. (You can also type code tags and by hand.)
   
2. Avoid adding color or different fonts and font size to your posts.
   Selective use of color to highlight a single word or phrase can be useful at times, but using color, in general, makes the forums harder to read, especially bright colors like red.
   
3. Be careful when you cut-and-paste, edit any odd characters and make sure all links are working property.

Thank You.

The UNIX and Linux Forums
Reply With Quote

Look like you are walking backward.
First you generate a file and now you try to fixit. Maybe you should generate the original file the way you like it from the beginning?

Thank you Danmero,

The problem is that I dont understand your code very well, I haven't been able to use change the lines on the fly, before they are sent to the file.

I will keep trying. Thank you!