Searching set of string from Live Running Logs

Shell Programming and Scripting
1

Hey just need one simple syntax to search for the string from the Live Running Logs. The strings are placed in a $infile & everytime the script should pull each string from $infile and should provide as an input for grepping from Live running logs on a rotational basis.

So here are the Contents of $inFile

87634863476
98349834987
98349873498
9JH83N3838
KJNE8302J3

How this can be used as an input for search string..

2

Put it in a file:

cat SearchStrings.txt
87634863476
98349834987
98349873498
9JH83N3838
KJNE8302J3

# And after
infile="SearchStrings.txt"
logFile="RunningLog.log"
while read mLine
do
       egrep ''"${mLine}"'' "${logFile}"
done < "${infile}"

Regards!

3

But I need to use different logs everytime eg: timelog.log other than runningLog.log . So what should be in that case. So the Log file should be defined by the user.

4

Ok! You can use something like:

echo "Please, enter the path and log file: "
read logFile

if [ ! -f "${logFile}" -o ! -r "${logFile}" ]
then
	echo "Invalid log file. It does not exist or is not readable."
	exit 1
fi

searchStrFile="SearchStrings.txt"
while read mLine
do
	egrep ''"${mLine}"'' "${logFile}"
done < "${searchStrFile}"
5

But i have log file in this type.

logFile1
logFile2
logFile3
logFile4
logFile5

But above syntax is only for one file at a time, but the string should be searched in all the logs simultaneously

Do i need to enter the range like this.

echo "Please, enter the path and log file: " read logFile[1-5]

 if [ ! -f "${logFile[1-5]}" -o ! -r "${logFile[1-5]}" ] then 	echo "Invalid log file. It does not exist or is not readable." 	exit 1 fi  searchStrFile="SearchStrings.txt" while read mLine do 	egrep ''"${mLine}"'' "${logFile}" done < "${searchStrFile}".
6

can you tell me if the log files you want to search in the same directory....
you can do something like this :- -

searchStrFile="SearchStrings.txt"
for file in `ls *`
do
echo "searching for the search pattern in the file $file"
while read mLine
do
echo "Searching for the line :-- $mLine"
egrep ''"${mLine}"'' "${file}"
done < "${searchStrFile}"
done
7

The previous syntax works fine for me , but my only concern is it should search in all the log files whose naming convention is same except the extension. But this files are under same directory.

8

you can change the above script in this way..

searchStrFile="SearchStrings.txt"
for file in `ls queryinfoservice_*`
do
echo "searching for the search pattern in the file $file"
while read mLine
do
echo "Searching for the line :-- $mLine"
egrep ''"${mLine}"'' "${file}"
done < "${searchStrFile}"
done
9

You don't really need the "ls queryinfoservice_*" inside the for loop, just:

searchStrFile="SearchStrings.txt"
searchFPattern="queryinfoservice_*"
for file in ${searchFPattern}
do
echo "searching for the search pattern in the file $file"
while read mLine
do
echo "Searching for the line :-- $mLine"
egrep ''"${mLine}"'' "${file}"
done < "${searchStrFile}"
done
10

Above syntax wont get me the result for multiple logs. Here is the pattern of logs wat i have.

queryFileService.log1 - queryFileService.log5
getURdetails.log1 - getURdetails.log5
...........
...........
............
nth.log1 - nth.log5

the search should not be an limited to only one log file. It should be user defined, but will exists in Log dir.

11

tell me one thing are there any specific logs that you dont want to process. if the user mention one particular file. you need all the files with that name and any extension .. then you can modify the above script like this... :-

echo "enter the name of the log u want to search"
read search_file
searchStrFile="SearchStrings.txt"
for file in `ls search_file*`
do
echo "searching for the search pattern in the file $file"
while read mLine
do
echo "Searching for the line :-- $mLine"
egrep ''"${mLine}"'' "${file}"
done < "${searchStrFile}"
done
12

This one will ask for the directory and file pattern to validate:

echo "Please, enter directory and log file name pattern: "
read searchDirFPattern

if [ ! -f "${searchDirFPattern}" ]
then
	echo "Invalid log file. It does not exist."
	exit 1
fi

searchStrFile="SearchStrings.txt"
for file in ${searchDirFPattern}
do
	echo "searching for the search pattern in the file $file"
	while read mLine
	do
		echo "Searching for the line :-- $mLine"
		egrep ''"${mLine}"'' "${file}"
	done < "${searchStrFile}"
done
13

Please ignore the previous instances of the log file format. Everything is fine with the script before and now also, but while entering the Path & File as an input to search for the string, I would rather provide in this way.

./search.sh
Please, enter directory and log file name pattern:
/logs/production/mlog/getinfo_LoggerGroup

From the above Input File am not providing any extension parameter like 1,2,3,4 or 5. Actually there are 5 instances of each log file.

The script should only search from above 5 files excluding zipped files which are also placed in the same dir, when I provide the input file name " getinfo_LoggerGroup " from the above syntax.

---------- Post updated 09-18-10 at 06:40 AM ---------- Previous update was 09-17-10 at 07:36 PM ----------

Can anybody help me to reorganize this script according to usage shown above.

14

All above method failed .. to get the string from multiple files. I just wanted to the script should read the File name String and search within all those files whose Pattern matches.

I have enclosed the attachment for File names, from which script should match the pattern and find in the LOG_DIR.

And here is the piece of Code,

#!/bin/bash

# Script name without path

BASENAME=${0##*/}

#Default Config & Log Path

myLocation=/home/rags234
myInst=${HOME}/config/application.lst
myLog="/logs/production/Application/$myInst"
myString="${HOME}/config/string

#Main Script

echo "Please, enter log file name pattern for SEARCH: ${myInst} "

read -r ${myInst}

if [ ! -f "${myLog}" ]
then
        echo "Invalid log file. It does not exist."
        exit 1
fi

#Searching the Pattern

for ${myInst} in ${myLog}
do
        echo "searching for the search pattern in the file $myInst"
        while read mLine
        do
                echo
                echo "============================================"
                echo "Searching for the line :-- $mLine"
                echo "============================================"
                echo
                egrep ''"${mLine}"'' "${myInst}"
        done <<EOF > "${myString}"
done;
15

Check the attached script and the comments on it.

Regards!