Making script run faster

Can someone help me edit the below script to make it run faster?

Shell: bash
OS: Linux Red Hat

The point of the script is to grab entire chunks of information that concerns the service "MEMORY_CHECK".

For each chunk, the beginning starts with "service {", and ends with "}".

I should point out that the status.log file is about 25MB.

grep -Pnw "MEMORY_CHECK" /apps/status.log | while read line

do

        NUMP=$(echo $line | awk -F":" '{print $1}')

        BEGINNINGA=$NUMP

        BEGINNING=$(NUM=1

        while [ $NUM -lt 70 ]

        do

                VAL=$NUMP

                MINUSED=$(echo $VAL $NUM | awk '{print $1 - $2}')

                DEFOU=$(sed -n ${MINUSED}p /apps/status.log | grep -Pv "#" | grep -P "service {")

                if [ ! -z "$DEFOU" ] ; then

                        echo "${MINUSED}:$DEFOU" | awk -F":" '{print $1}'

                        break

                fi

                NUM=$(( $NUM + 1 ))

        done)

##############################

        END=$(NUM=1

        while [ $NUM -lt 70 ]

        do

                VAL=$NUMP

                MINUSED=$(echo $VAL $NUM | awk '{print $1 + $2}')

                DEFOU=$(sed -n ${MINUSED}p /apps/status.log | grep -Pv "#" | grep -P "}")

                if [ ! -z "$DEFOU" ] ; then

                        echo "${MINUSED}:$DEFOU" | awk -F":" '{print $1}'

                        break

                fi

                NUM=$(( $NUM + 1 ))

        done)


        echo "==================================================="
        echo ""

        BAD=$(sed -n ${BEGINNING},${END}p/apps/status.log  | egrep "CRITICAL:|UNKNOWN:")

        if [ ! -z "$BAD" ] ; then

                sed -n ${BEGINNING},${END}p /apps/status.log  | egrep "host_name=|check_command=|^plugin_output="

        fi

        echo ""
        echo "==================================================="

done

You're running grep, awk, and sed hundreds of thousands of times to process thousands of lines. I think your script may need a rewrite. You could probably do it all in one awk instance.

Can you show the input data you have, and the output you want?

here's the input data. Notice the below is what i refer to as a chunk. in this particular case, the chunk below belongs to the service called "MEMORY_CHECK".

so there's a file that contains several of these chunks for several services.

lets say the file is called status.log. for processing, I want to pull out all chunks called "service_description=MEMORY_CHECK" and then pull out other attributes of each of the chunk as needed.

attributes can be, i.e what is the name of the hostname that this MEMORY_CHECK is on? in this case, the host name would be "sky.log.net".

what is the plugin output, in this case, it would be "CRITICAL: Process was not found".

hope this helps.

service {
host_name=sky.log.net
service_description=MEMORY_CHECK
modified_attributes=1
check_command=blah!blah!blah
check_period=24x7
notification_period=24x7
event_handler=
has_been_checked=1
check_execution_time=0.453
check_latency=1.701
check_type=0
current_state=2
last_state=2
last_hard_state=2
last_event_id=2522339
current_event_id=2523376
current_problem_id=1127556
last_problem_id=0
current_attempt=1
max_attempts=1
normal_check_interval=30.000000
retry_check_interval=2.000000
state_type=1
last_state_change=1335466482
last_hard_state_change=1335466482
last_time_ok=1334123115
last_time_warning=0
last_time_unknown=1335465427
last_time_critical=1336750629
plugin_output=CRITICAL:  Process was not found
long_plugin_output=
performance_data=
last_check=1336750629
next_check=1336752429
check_options=0
notified_on_unknown=1
notified_on_warning=0
notified_on_critical=1
current_notification_number=364
current_notification_id=2530346
last_notification=0
notifications_enabled=0
active_checks_enabled=1
passive_checks_enabled=1
event_handler_enabled=1
problem_has_been_acknowledged=0
acknowledgement_type=0
flap_detection_enabled=0
failure_prediction_enabled=1
process_performance_data=1
obsess_over_service=1
is_flapping=0
percent_state_change=0.00
check_flapping_recovery_notification=0
state_history=2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2
}

Yes, and what output do you want now? "Pull out" isn't too specific, what does the data you want actually look like?

Here's a start at least, which only prints service { sections containing service_description=MEMORY_CHECK :

$ cat memorycheck.awk
BEGIN { FS="\n"; RS="}\n"; ORS="}\n\n"; }

{
        for(X in D) delete D[X];

        for(N=2; N<=NF; N++)
        {
                split($N, A, "=");
                D[A[1]]=A[2];
        }
} D["service_description"]=="MEMORY_CHECK"

$ awk -f memorycheck.awk data

service {
host_name=sky.log.net
service_description=MEMORY_CHECK
modified_attributes=1
check_command=blah!blah!blah
check_period=24x7
notification_period=24x7
event_handler=
has_been_checked=1
check_execution_time=0.453
check_latency=1.701
check_type=0
current_state=2
last_state=2
last_hard_state=2
last_event_id=2522339
current_event_id=2523376
current_problem_id=1127556
last_problem_id=0
current_attempt=1
max_attempts=1
normal_check_interval=30.000000
retry_check_interval=2.000000
state_type=1
last_state_change=1335466482
last_hard_state_change=1335466482
last_time_ok=1334123115
last_time_warning=0
last_time_unknown=1335465427
last_time_critical=1336750629
plugin_output=CRITICAL:  Process was not found
long_plugin_output=
performance_data=
last_check=1336750629
next_check=1336752429
check_options=0
notified_on_unknown=1
notified_on_warning=0
notified_on_critical=1
current_notification_number=364
current_notification_id=2530346
last_notification=0
notifications_enabled=0
active_checks_enabled=1
passive_checks_enabled=1
event_handler_enabled=1
problem_has_been_acknowledged=0
acknowledgement_type=0
flap_detection_enabled=0
failure_prediction_enabled=1
process_performance_data=1
obsess_over_service=1
is_flapping=0
percent_state_change=0.00
check_flapping_recovery_notification=0
state_history=2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2
}

$

If awk doesn't work for you, try nawk.

All the various fields for each section should be available inside the D array inside awk, for you to do with as you please.

If you could actually tell me what you wanted to do with them, all the better.

you are a genius.

just one last question, i don't want to have to run the script like this:

awk -f memorycheck.awk data

i modified the code and put it all in one script, but it's not working:

awk BEGIN { FS="\n"; RS="}\n"; ORS="}\n\n"; }

{
        for(X in D) delete D[X];

        for(N=2; N<=NF; N++)
        {
                split($N, A, "=");
                D[A[1]]=A[2];
        }
} D["service_description"]=="MEMORY_CHECK" status.log

i get the following errors:

./script: line 1: syntax error near unexpected token `}'
./script: line 1: `awk BEGIN { FS="\n"; RS="}\n"; ORS="}\n\n"; }'

and also, i want to get (from each chunk), information about the "host_name=" and "plugin_output="

how do i specify that in the script? i want to be able to specify in the script to only output the MEMORY_CHECK information about so and so host.

I'm not being cranky about this - but you do realize that some of us have possibly been doing this stuff since before you were born. So it is at least remotely possible that we know what are doing, in a collective sense. I started UNIX in 1975.

Ok?

So - could you please find the time to tell us what you are trying to do - not what you think you should do to get to your goal? We need a description of your goal, devoid of your idea of how to do it. Thanks. It will help all of us in the long run.

If you want to put it in a shell script, you have to use single quotes:

#!/bin/bash

awk 'BEGIN { FS="\n"; RS="}\n"; ORS="}\n\n"; }

{
        for(X in D) delete D[X];

        for(N=2; N<=NF; N++)
        {
                split($N, A, "=");
                D[A[1]]=A[2];
        }
}
D["service_description"]==pat' pat=$2 $1

Then you can call it with

./script.sh status.log MEMORY_CHECK

The last line,

D["service_description"]==pat

filters the chunk and prints only the chunk that contains

service_description=<pattern>

So you could make a compound statement, with logical AND (&&) or OR (||) operators:

...
D["service_description"]==pat && D["host"]==h ' h=$3 pat=$2 $1

Then the script you'd invoke like

./script.sh status.log MEMORY_CHECK <hostname>

It will still print the whole chunk though. If you just want to see the line that contains "host", pipe the output (selected chunk) to grep.

But admittedly, the cranky old guy is right that you should be as specific as possible in what you are trying to achieve, in order to get fast and accurate help. It would save us time and enable us to help more people.

hey guys, thanks for your help. i thought i gave as much information as would be needed in my previous posts. but i apologize if that wasn't enough. i didn't want to overindulge you with too much explanation and cause probable confusion.

but here is what i'm trying to do.

the chunk that i posted is a chunk that exist in a giant file for a number of servers.

so, i have about 2500 servers being monitored. and EACH of those servers have chunks in the status.log file. chunk(s) is plural because, for each server, there can be more than one chunk. matter of fact, there is usually at least 4 chunks for each server. each of those 4 chunks represent service_descriptions (i.e., CPU_CHECK, MEMORY_CHECK, DISK_CHECK etc). so you can imagine how huge that file is.

now, my task is to grab information on ANY host out of the 2500 from the status.log file. the problem is, when you have a file that is big with information of 2500 servers stored in it, processing the file becomes sort of a nightmare. and response is very slow.

in the script below, i attempted to do precisely what i need, i hope you guys can offer suggestions:

#!/bin/bash

cat Servers.txt | while read server

do

OUTPUT=$(

awk 'BEGIN { FS="\n"; RS="}\n"; ORS="}\n\n"; }

{
        for(X in D) delete D[X];

        for(N=2; N<=NF; N++)
        {
                split($N, A, "=");
                D[A[1]]=A[2];
        }
}

D["service_description"]==pat && D["host_name"]==h ' h=$server pat=$2 $1 | egrep "^plugin_output"

)

echo "$server  ==  $OUTPUT"

done

so i call this script like this:

./script status.log MEMORY_CHECK

so in essence, what i'm saying through this script is that, for EACH server in the list provided in Servers.txt, I want you to pull out the chunk for "service_description" titled MEMORY_CHECK. and from the output, i want you to only grab out information about plugin_output.

i ran this just now on a list of about 200 server and while it is working, it is taking forever to complete.

You are processing the whole logfile for each server separately, that's what hinders you.
Try this:

#!/bin/bash

awk 'BEGIN {
  while((getline < "server.list")>0)
     S[$0]

  FS="\n"; RS="}\n"; ORS="}\n";
}

/service_description=MEMORY_CHECK/ {

  for(X in D) delete D[X];

  for(N=2; N<=NF; N++)
  {
       split($N, A, "=");
       D[A[1]]=A[2];
  }

  if (D["host_name"] in S)
       printf("%20s -- %50s\n", D["host_name"], D["plugin_output"])

}' $1

Stash all the host names in a text file, one host per line, save it as "server.list", then call this like

./script.sh status.log

Wow. this actually seems to be super fast. however, it doesn't go through the entire file. it only spat out arbitrary servers. out of 200 servers, it only spat out 8 servers, and there should be way more than that. and also it provides limited output for the "plugin_output". it didn't show the entire output of the plugin_output.

Your feedback is (once again) not very constructive. Please:

1. give an example of one hostname that it doesn't catch, post the chunk that you are missing. What do you expect to get?

2. Furthermore, make sure your server list contains one server per line, with no whitespaces, and no extra characters (e.g. if you created this on windows system)

3. Try to understand what it does. This way you can learn, and adjust to your particular needs:

• the script processes only the chunks with

service_description=MEMORY_CHECK

• and it only prints the host name and the part of plugin_output line that is after the = sign.

There are different servers in the file. and yes, it is one server per line with no extra spaces.

Lets assume in the file containing the list of servers, there are servers with different naming schemes. some servers are named apples1, apples2 etc.
others are named oranges1, oranges2 etc. and some are named serverA, serverB, as is shown below.

each one of all these servers has a service_description on it for MEMORY_CHECK. however, when i run the script, it doesn't grab all these servers.
it grabs random servers, examples of which is shown below. the random servers that it did grab, it only shows part of the "plugin output" instead of everything.

when i run the script, i get this output:

serverA --                                          OK: Used
serverB --                                          OK: Used
serverC --                                          OK: Used
serverD --                                          OK: Used
serverE --                                          OK: Used
serverF --                                          OK: Used
serverG --                                          OK: Used
serverH --                                          OK: Used

the "OK: Used", should really show the complete output which is:

OK: Used = [ 394.941 MB ], System = [ 11.8867 GB ], Used = [ 3.24464% ], Available = [ 11.501 GB ], Cached = [ 11.248 GB ].

so in other words, when i run the script, i should get something like this for each server:

serverA --                                          OK: Used = [ 394.941 MB ], System = [ 11.8867 GB ], Used = [ 3.24464% ], Available = [ 11.501 GB ], Cached = [ 11.248 GB ].

OK, well, that is progress.
Notice these two lines?

       
      split($N, A, "="); 
      D[A[1]]=A[2];

This splits the line (stored in $N) on '=' and stores it in array A.
The second line stores the second field only A[2].
Since nobody mentioned before that the lines could have multiple equal signs, it was assumed that lines were in the form

label=value

where the value would not contain the '=' (a very resonable assumption).

So, you need to append the rest of the fields, like e.g.:

awk 'BEGIN {
  while((getline < "server.list")>0)
     S[$0]

  FS="\n"; RS="}\n"
}

/service_description=MEMORY_CHECK/ {

  for(X in D) delete D[X];

  for(N=2; N<=NF; N++)
  {
       split($N, A, "=");
       D[A[1]] = A[2]
       i = 3;
       while (i in A) 
          D[A[1]] = D[A[1]] "=" A[i++];
  }

  if (D["host_name"] in S) 
       printf("%20s -- %50s\n", D["host_name"], D["plugin_output"])

}' $1
 

Now did you read the code and try to think how it works? The order is not random, at all. It processes one chunk at a time, and looks if the particular chunk's hostname is in the server list.
If you insist to process them in a fixed order, by-server, you would have to do what you did before, scan the log once for each server.
So I suggest you take the order that is given by the log file, and sort the results to get them in the order you want.

Your problem is that your input file is read two often.
The solution is to parse it in one stroke.
Most speedy are awk then perl. Here is a pure shell script, just to demonstrate that the algorithm makes the difference:

#!/bin/sh
#set -x
ruler="==================================================="
nl="
"
inblock=0

while read line
do
  case $line in
  "service_description=MEMORY_CHECK")
    found1=1
    ;;
  *"CRITICAL:"*|*"UNKNOWN:"*)
    found2=1
    ;;
  'service{'*|'service {'*)
    inblock=1
    found1=0
    found2=0
    str=""
    ;;
  '}'*)
    inblock=0
    if [ $found1 -ne 0 -a $found2 -ne 0 ]
    then
      printf "%s\n%s\n\n%s\n" "$ruler" "$str" "$ruler"
    fi
    ;;
  host_name=*|check_command=*|plugin_output=*)
    str="${str}${nl}${line}"
    ;;
  esac 
done < status.log

i've been playing with this code for a while but cant seem to get a handle on a seemingly minor issue.

the one thing i didn't notice about the status.log file is that there are several other chunks in it that do not have the beginning tags of "service {" and ending tags of "}"

some chunks start with "servicecomment {" and end with "}". others start with "contact {" and ends with "}".

however, the only chunks im interested in are the ones that start with "service {" and end with "}"

how can i specify the "service {" in your code?

your code works PERFECTLY. i just dont know how to modify it. i dont mean to keep pestering about this. i'm just so close to completing my task.

---------- Post updated at 09:35 PM ---------- Previous update was at 08:56 PM ----------

i got it. the fix was to add this to the script:

/service_description=MEMORY_CHECK/ && /service {/

thanks to every one!