Calculate avg response time on hourly basis

Shell Programming and Scripting
1

Hi,

I am trying to calculate avg response time on hourly basis from the log file which has millions of records.

As of now I am trying with creating temp file which will have lines with unique id and start time and end time and after that another script will run on this temp file to calcualte avg response time.
Reason I posted this ques in forum becuase this way takes more than an hour to create temp file.

Is there any way we can do it quicker?
Note:These UNIQID is not coming in sequence.

2012-06-04 13:04:19,324 UNIQID1
2012-06-04 13:04:20,120 UNIQID1
2012-06-04 13:05:19,324 UNIQID2
2012-06-04 13:06:20,120 UNIQID2
2012-06-04 13:07:19,324 UNIQID3
2012-06-04 13:08:20,120 UNIQID3
2012-06-04 13:08:49,324 UNIQID4
2012-06-04 13:09:50,120 UNIQID4
2

Whats your current script which is taking hours of time? May be we can try to finetune the same for you :slight_smile:

3 
for uid in ${uids}; do
count=`grep "$uid" test.log|wc -l`
if [ "${count}" -ne "0" ]; then
unique_uids[counter]="$uid"
#echo "${unique_uids[counter]}"
let counter=counter+1
fi
done
echo ${unique_uids[@]}
echo $counter
echo " Unique No:" ${#unique_uids[@]}
echo uid StartTime EndTime" > $log
for unique_uids in ${unique_uids[@]} ; do
 
responseTime=`cat $i|grep "${unique_uids}" |awk '{split($2,Arr,":|,"); print Arr[1]*3600000+Arr[2]*60000+Arr[3]*1000+Arr[4]}'|sort -n`
echo $unique_uids $responseTime >> $log
done

Output will be like this

UID Start time end time
UNIQID1 13:04:19,324 13:04:20,120

---------- Post updated at 07:12 AM ---------- Previous update was at 05:33 AM ----------

pls let me know if something in script is not clear. The first for loop is find unique no of ids and then grepping those uniq ids to find start and end time.

Thanks for you time!!

4

This

responseTime=`cat $i|grep "${unique_uids}" |awk '{split($2,Arr,":|,"); print Arr[1]*3600000+Arr[2]*60000+Arr[3]*1000+Arr[4]}'|sort -n`

may be some rearranged

responseTime=$(awk '$0~u {split($1,Arr,":|,"); print 1000*(Arr[1]*3600+Arr[2]*60+Arr[3])+Arr[4]}' u="${unique_uids}" $i | sort -n)
5

Thanks for looking into this,

This exactly work like my previous code, how ever the pain area to get avg response time for each hour.

like in 24 hr format 00,01,02...23 for each hour avg responsetime would be assume if 10 hist then total response time( end time - start time) sum of response times for 10 hits/ 10

Thanks,

6

Hi Jotne, Can you pls explain me what this common does. inside awk $0~u

7

True if the content of $0 match the content of variable u

8

It search all line if it contain text stored in variable ${unique_uids}
Same as your grep here: grep "${unique_uids}"