Grep string from logs of last 1 hour on files of 2 different servers and calculate count

Shell Programming and Scripting
1

Hi,
I am trying to grep a particular string from the files of 2 different servers without copying and calculate the total count of its occurence on both files.
File structure is same on both servers and for reference as follows:

27-Aug-2010 10:04:30,601|919122874903|phtunes_app|1282243292627|NotifySmsReception|DMGenerateLogInterceptor - ExternalTransactionID:SDP-DM-26713018, TransactionStatus:Requested
27-Aug-2010 10:05:30,601|919122874903|phtunes_app|1282243292627|NotifySmsReception|MaskingUnMaskingInterceptor - msisdn before masking 9122874903
27-Aug-2010 16:33:30,627|919122874903|phtunes_app|1282243292627|NotifySmsReception|MaskingUnMaskingInterceptor - msisdn after masking BJ#13340708
27-Aug-2010 16:34:30,637|BJ#13340708|phtunes_app|1282243292627|NotifySmsReception|CP URL:http://172.30.24.52/unitech_sms/unitechsms.php, AppInstanceIdhtunes_app, Keyword:busnews, SID:tel:BJ%2313340708, TransactionID:SDP-DM-26713018

---------- Post updated at 03:11 PM ---------- Previous update was at 03:09 PM ----------

Please consider this example for grep string from logs of last 15 minutes too.

---------- Post updated at 03:41 PM ---------- Previous update was at 03:11 PM ----------

Find the code below which i have made but getting error.
The code checks the last 15 minutes logs and search the string and get the total count.Can anybody correct the following code or provide me the optimized way:

bin/bash

to=`date +"%d-%b-%Y %T"`
echo $to
let from_in_seconds=`date +%s`-900
from=`date -d @$from_in_seconds +"%d-%b-%Y %T"`
echo $from

shortcodes=( "56882" "58585" "58888" "57575" "57677" );
for shortcode in ${shortcodes[@]}
do
    count=0
    sh_count=`awk '$0>=from && $0<=to' from="$from" to="$to" /opt/bea/ocsg_4.1/logs/customlogs/App_OP.log | grep "ShortCode=tel:${shortcode}" | wc -l`
    count=`expr $count + $sh_count`
    sh_count2=`ssh -n smehan@10.0.0.1 "awk '$0>=from && $0<=to' from=\"$from\" to=\"$to\" /opt/bea/ocsg_4.1/logs/customlogs/App_OP.log | grep \"ShortCode=tel:${shortcode}\" | wc -l"`
    count=`expr $count + $sh_count2`
    echo "${shortcode} : "$count
done
2

post the errors. also please confirm that the first line of your scripts is

#!/bin/bash
3

Hi Frank,

Yes the first line is #!/bin/bash in code.
Error i am getting is grepstring.sh is not found on line where i am calling command on server 2.

4

add these lines to the top of the script(but after the first line) and then post the entire output. the command your running remotely is kind of complex to troubleshoot because of the variables and quoting. can you put the logic in a script on the remote side and then just execute the script?

set -x
set -u

another suggestion - use

grep -c

instead of

|wc -l

you could easily remove the grep and place use the logic in awk to streamline even further.

5

Hi frank,

Check my output below and suggest the changes that have to be done or more optimized way of doing that:

+ set -u
++ date '+%d-%b-%Y %T'
+ to='06-Sep-2010 10:46:22'
+ echo 06-Sep-2010 10:46:22
06-Sep-2010 10:46:22
++ date +%s
+ let from_in_seconds=1283750182-900
++ date -d @1283749282 '+%d-%b-%Y %T'
+ from='06-Sep-2010 10:31:22'
+ echo 06-Sep-2010 10:31:22
06-Sep-2010 10:31:22
+ shortcodes=("56882")
+ for shortcode in '${shortcodes[@]}'
+ count=0
++ awk '$0>=from && $0<=to' 'from=06-Sep-2010 10:31:22' 'to=06-Sep-2010 10:46:22' /opt/bea/ocsg_4.1/logs/customlogs/App_OP.log
++ grep -c ShortCode=tel:56882
+ sh_count=0
++ expr 0 + 0
+ count=0
++ ssh -n smehan@172.30.16.225
+ grep -c ShortCode=tel:56882
Pseudo-terminal will not be allocated because stdin is not a terminal.
+ sh_count2=awk
+ '$0>=from && $0<=to' 'from=06-Sep-2010 10:31:22' 'to=06-Sep-2010 10:46:22' /opt/bea/ocsg_4.1/logs/customlogs/App_OP.log
./test_mo_count2: line 17: $0>=from && $0<=to: command not found
0
./test_mo_count2: line 18: sh_count2: unbound variable
+ count=
+ echo '56882 : '
56882 :