Substitute grep command at run time

Shell Programming and Scripting
1

HI
I am trying to use the following code in the shell script (using grep)

usage()
{
  echo "Usage: ./$0 <file name> <interval> <pattern>"
} 

METRICS_FILE=$1
INTERVAL=$2
PATTERN="$3" 
..

if [ "${PATTERN}" == "" ]
then
 PATTERN="grep Gx"
fi

	COUNT=`cat ${METRICS_FILE} | "${PATTERN}" |egrep "${start_tim}|${end_tim}" |awk -F"Count=" '{print $2}'|cut -d',' -f1 |perl -lne '$sum += $_ } { print $sum'`

..

The above is working fine in case I do not provide any argument for arg no 3, as in that case the hard-coded string "PATTERN="grep Gx"", is used.

But in case i provide argument 3 as "grep gx|grep -v RAR"

I am getting error.:

Please suggest.

2

Firstly, looks like the end of perl command is missing curly bracers in the end.
You should specify your operating system and shell used.

I cannot replicate behavior you are experiencing neither working or non working example on debian linux, using bash shell.

As for code, well i'm sure it could be written better, if you provide the relevant data to parse, and script expected input/output.

A PATTERN should be a string to match not a grep pipe grep command(s) inside a shell variable.
This is just wrong on so many levels and will not work as you expect or at all.

So my suggestion would be a complete rewrite :smiley:

Also, for instance, you can specify a default value using shell builtin.

PATTERN=${1:-Gx}
echo $PATTERN

Meaning if $1 is not inputted or defined, it will be Gx otherwise it will be what is inputted/defined.

Hope that helps
Regards
Peasant.

3

In Posix shell, bash, ksh and zsh, splitting the command line into the individual commands, takes place before parameter expansion. Hence, the shell doesn't see the pipe inside your pattern.

BTW, with this type of questions, you should always state which shell you are using.

1 Like
4

Its a bash shell.
Any ways to make it work.

------ Post updated at 12:05 PM ------

Thanks for the prompt reply..
As stated the script is working fine if I do not give 3rd argument, this means the syntax which you are pointing is not correct, is not the case.

Here is the complete code:

#!/bin/bash

usage()
{
  echo "Usage: ./$0 <metrics file name> <interval> <pattern>"
}  

METRICS_FILE=$1
INTERVAL=$2
PATTERN="$3"

if [ "${METRICS_FILE}" == "" ] || [ "${INTERVAL}" == "" ] 
then
	echo "Input not sufficient"
	usage
	exit
fi

if [ "${PATTERN}" == "" ]
then
 PATTERN="grep Gx"
fi

cat ${METRICS_FILE} |cut -d',' -f2,4|uniq > interval.txt
DUR=`expr ${INTERVAL} \* 60`
while read line
do
	
	start_tim=`echo ${line} |cut -d',' -f1`
	end_tim=`echo ${line} |cut -d',' -f2`
	
	COUNT=$(cat ${METRICS_FILE} | ${PATTERN} |egrep "${start_tim}|${end_tim}" |awk -F"Count=" '{print $2}'|cut -d',' -f1 |perl -lne '$sum += $_ } { print $sum')

	if [ "${COUNT}" == "" ]
	then
		continue;
	fi	
	RATE=`awk "BEGIN {print ${COUNT}/${DUR}}"`  	
#	echo "Interval: $line"	
	echo "COUNT: [${COUNT}] RATE: [${RATE}] [${start_tim}] [${end_tim}]"
	echo
done < interval.txt
rm -rf interval.txt 2>/dev/null

Sample Input File: Paste the contents in m.csv.


Start Time In MS=1532947500004,Start Time Local=Mon Jul 30 12:45:00 CEST 2018,End Time In MS=1532948400003,End Time Local=Mon Jul 30 13:00:00 CEST 2018,Site=site1,Group=Diameter,Application=Gx,Command=RAR,Destination Host=pcrf1.vVOLTE5MIRM.site1,Destination Realm=EPC.MNC010.MCC222.3GPPNETWORK.ORG,Egress Peer Origin Host=csb.vVOLTE5MIRM.site1,Egress Peer Origin Realm=vVOLTE5MIRM.site1,Ingress Peer Origin Host=c11-10-214-191-34-dcc-prf-mas-44,Ingress Peer Origin Realm=vodafone.it,Origin Host=c11-10-214-191-34-dcc-prf-mas-44,Origin Realm=vodafone.it,Outbound Message Processing=No Value,Result=DIAMETER_SUCCESS,Role=Routing Agent,Average Latency=6.621212121212121
Start Time In MS=1532947500004,Start Time Local=Mon Jul 30 12:45:00 CEST 2018,End Time In MS=1532948400003,End Time Local=Mon Jul 30 13:00:00 CEST 2018,Site=site1,Group=Diameter,Application=Gx,Command=CCR,Destination Host=pcrf1.vVOLTE5MIRM.site1,Destination Realm=EPC.MNC010.MCC222.3GPPNETWORK.ORG,Egress Peer Origin Host=csb.vVOLTE5MIRM.site1,Egress Peer Origin Realm=vVOLTE5MIRM.site1,Ingress Peer Origin Host=c11-10-214-191-34-dcc-prf-mas-44,Ingress Peer Origin Realm=vodafone.it,Origin Host=c11-10-214-191-34-dcc-prf-mas-44,Origin Realm=vodafone.it,Outbound Message Processing=No Value,Result=DIAMETER_SUCCESS,Role=Routing Agent,Average Latency=6.621212121212121

And run it as:

./r.sh m.csv 15
COUNT: [0] RATE: [0] [Start Time Local=Mon Jul 30 12:45:00 CEST 2018] [End Time Local=Mon Jul 30 13:00:00 CEST 2018]

To generate error:

[root@dsc02-1 vf]# ./r.sh m.csv 15 "grep Gx|grep -v RAR"
grep: RAR: No such file or directory
5

Maybe something in this way (not tested):

alias PATTERN="${3:-grep whatever you want here}"

.....

COUNT=$(PATTERN|egrep "${start_tim}|${end_tim}" | awk ... etc etc ...)
6

I think I have already checked this, here is the execution results:

[root@dsc02-1 vf]# ./r.sh metrics.csv.20180730_13_00 15 "grep Gx|grep -v RAR"
+ METRICS_FILE=metrics.csv.20180730_13_00
+ INTERVAL=15
+ PATTERN='grep Gx|grep -v RAR'
+ '[' metrics.csv.20180730_13_00 == '' ']'
+ '[' 15 == '' ']'
+ '[' 'grep Gx|grep -v RAR' == '' ']'
+ cat metrics.csv.20180730_13_00
+ cut -d, -f2,4
+ uniq
++ expr 15 '*' 60
+ DUR=900
+ read line
++ echo Start Time Local=Mon Jul 30 12:45:00 CEST 2018,End Time Local=Mon Jul 30 13:00:00 CEST 2018
++ cut -d, -f1
+ start_tim='Start Time Local=Mon Jul 30 12:45:00 CEST 2018'
++ echo Start Time Local=Mon Jul 30 12:45:00 CEST 2018,End Time Local=Mon Jul 30 13:00:00 CEST 2018
++ cut -d, -f2
+ end_tim='End Time Local=Mon Jul 30 13:00:00 CEST 2018'
++ cat metrics.csv.20180730_13_00
++ grep 'Gx|grep' -v RAR
++ egrep 'Start Time Local=Mon Jul 30 12:45:00 CEST 2018|End Time Local=Mon Jul 30 13:00:00 CEST 2018'
++ awk -FCount= '{print $2}'
++ cut -d, -f1
grep: RAR: No such file or directory
++ perl -lne '$sum += $_ } { print $sum'
+ COUNT=
+ '[' '' == '' ']'
+ continue
+ read line
+ rm -rf interval.txt

What I used:

#!/bin/bash
set -x
usage()
{
  echo "Usage: ./$0 <metrics file name> <interval> <pattern>"
}  

METRICS_FILE=$1
INTERVAL=$2
#PATTERN="$3"
PATTERN="${3:-grep Gx}"

if [ "${METRICS_FILE}" == "" ] || [ "${INTERVAL}" == "" ] 
then
	echo "Input not sufficient"
	usage
	exit
fi

if [ "${PATTERN}" == "" ]
then
 PATTERN="grep Gx"
fi

cat ${METRICS_FILE} |cut -d',' -f2,4|uniq > interval.txt
DUR=`expr ${INTERVAL} \* 60`
while read line
do
	
	start_tim=`echo ${line} |cut -d',' -f1`
	end_tim=`echo ${line} |cut -d',' -f2`
	
	COUNT=$(cat ${METRICS_FILE} | ${PATTERN} |egrep "${start_tim}|${end_tim}" |awk -F"Count=" '{print $2}'|cut -d',' -f1 |perl -lne '$sum += $_ } { print $sum')

	if [ "${COUNT}" == "" ]
	then
		continue;
	fi	
	RATE=`awk "BEGIN {print ${COUNT}/${DUR}}"`  	
#	echo "Interval: $line"	
	echo "COUNT: [${COUNT}] RATE: [${RATE}] [${start_tim}] [${end_tim}]"
	echo
done < interval.txt
rm -rf interval.txt 2>/dev/null
7

As I already explained, this doesn't work. Why don't you try the code which I have suggested?

8

I didn't get what you suggested..

alias PATTERN="${3:-grep whatever you want here}"

.....

COUNT=$(PATTERN|egrep "${start_tim}|${end_tim}" | awk ... etc etc ...)

While I need

COUNT=$(cat ${METRICS_FILE} | ${PATTERN} |egrep "${start_tim}|${end_tim}" |awk -F"Count=" '{print $2}'|cut -d',' -f1 |perl -lne '$sum += $_ } { print $sum')
9

Once more:

  • Since interpolation of the variable does not work, as I explained, I suggested that you model your "pattern" not by a variable named PATTERN, but by an alias named PATTERN.

  • Also note that in my proposal, I suggested to drop the unnecessary cat command and replaced the depecated backquotes for process substitution, which you had used, by $(....). These changes don't affect the functionality, though.

1 Like