How to read each file grep a value from that?

darling · October 23, 2017, 12:44pm

Hi Team,

in /tmp folder i have thousands of log files i want to read each file and grep a value called "Calling This".

Each logfile name is different but it ends with .log.

How can i achieve this?

Please excuse if i did any mistake by not following forum standards. I will surely follow if i insert any code.

joeyg · October 23, 2017, 12:51pm

Please provide information on your attempts to resolve.

Until so, we will refrain from sharing any guidance.

The purpose of this Board is the assist users in solving their problems. We are not a coding service. Further, we are not a homework service - and sometimes posts appear to be an attempt to have someone solve a school assignment.

Finally, we urge all members of the Forum to NOT post a solution to this question until effort to resolve is demonstrated.

darling · October 23, 2017, 1:01pm

Below are the steps i did

while read filename; do grep -Hi 'Calling ' done

RavinderSingh13 · October 23, 2017, 1:10pm

Hello darling,

Thanks for showing us your efforts, as mentioned by Joeyg please always spend sometime while posting any question, provide more information(eg--> your O.S name, sample Input_file and expected output_file in code tags).
If you want to search a specific string in all the .txt files and want to print only those file names which have them inside it, then following may help you in same.

find /tmp -type f -name "*.log" -exec grep -l "Calling this" {} \+

kindly give it a try and do let us know how it goes then(in detailed manner please).

Thanks,
R. Singh

darling · October 23, 2017, 1:13pm

ravindersingh13:

Hello darling,

Thanks for showing us your efforts, as mentioned by Joeyg please always spend sometime while posting any question, provide more information(eg--> your O.S name, sample Input_file and expected output_file in code tags).
If you want to search a specific string in all the .txt files and want to print only those file names which have them inside it, then following may help you in same.
find /tmp -type f -name "*.log" -exec grep -l "Calling this" {} \+
kindly give it a try and do let us know how it goes then(in detailed manner please).

Thanks,
R. Singh

Thanks Singh,

It is giving me the output of the files.

command

find /tmp/S -type f -name "*.log" -exec grep -l "Calling appalert" {} \+

/tmp/S/BmmRaiseAlert.sh.201710221859.31036.logfile.log
/tmp/S/BmmRaiseAlert.sh.201710131507.32473.logfile.log
/tmp/S/BmmRaiseAlert.sh.201710190037.17027.logfile.log

Linux - OS

Don_Cragun · October 23, 2017, 2:14pm

If all of the files are in a single directory (not spread out among subdirectories), you could try the following first:

cd /tmp
grep -FHi 'Calling appalert' *.log

or, if you just want the names of files containing that string (with a case insensitive match):

cd /tmp
grep -Fli 'Calling appalert' *.log

These might file with "argument list too long" errors depending on how many thousands of files you have, but it is easy to try it. And, if it works, it will hundreds or thousands of times faster that performing a grep on each file individually. If they won't work, you'll know quickly.

Note that I added a -F option to grep . Searching for fixed strings (even if you're doing a case-insensitive match), is faster than going through the extra complexity of matching regular expressions.

If all of the log entries use the same case, dropping the -i option will also make it run faster.

MadeInGermany · October 23, 2017, 2:23pm

The grep -l "Calling appalert" gives the pathnames of the files that have a line with Calling appalert .
Do you want something else?
Then give an example: a sample of an input file with a matching and a non-matching line, and the expected output.

darling · October 24, 2017, 4:33am

Hi,

Below is the content from my file..


BmmRaiseAlert.sh::308: AlertConfigResult [MatchedOn=033A,Appalert=A60107,Description=eRIC for MWW not processed messages for 5 minutes]
BmmRaiseAlert.sh::320: Calling BuildXHTMLForBmtm.sh with Bmm File [/opt/BMC/BMTM/RFRAlertingIntegration/BmmOutputFiles/BMMEventOutput.1508657897.215.812]
BmmRaiseAlert.sh::323: Call to BuildXHTMLForBmtm.sh complete
BmmRaiseAlert.sh::365: Calling appalert with args [ -l 3 -a MQ -e A60107 -f gbahevpl41.krf.express.RFR -u "http://messagemonitor.krf.express.RFR/BMTM/alerts/BMMEventOutput.1508657897.215.812.html" -m "gbahevpl41.krf.express.RFR:QPAHECL1:QTW1.BOQ.109 ==> eRIC for MWW not processed messages for 5 minutes"]
BmmRaiseAlert.sh::367: Call to appalert completed
BmmRaiseAlert.sh::369: Completed successfully

I want out from each file with file name after grepping the value into a single file.. For example

Name of first log file ->  line starting with "AlertConfigResult" and starting with "Calling appalert" and "last line"

BmmRaiseAlert.sh.201710220838.6155.logfile.log ->BmmRaiseAlert.sh::308: AlertConfigResult [MatchedOn=033A,Appalert=A60107,Description=eRIC for MWW not processed messages for 5 minutes]  

Calling appalert with args [ -l 3 -a MQ -e A60107 -f gbahevpl41.krf.express.RFR -u "http://messagemonitor.krf.express.RFR/BMTM/alerts/BMMEventOutput.1508657897.215.812.html" -m "gbahevpl41.krf.express.RFR:QPAHECL1:QTW1.BOQ.109 ==> eRIC for MWW not processed messages for 5 minutes"]


BmmRaiseAlert.sh::369: Completed successfully


Name of second log file ->  line starting with AlertConfigResult and Calling appalert and last line

Name of n log file ->  line starting with AlertConfigResult and Calling appalert and last line

we can ignore first 22 characters like "BmmRaiseAlert.sh::369:"

---------- Post updated at 03:33 AM ---------- Previous update was at 03:22 AM ----------

i tried below

cat BmmRaiseAlert.sh.201710220838.6155.logfile.log| egrep 'Calling appalert|Completed'

BmmRaiseAlert.sh::365: Calling appalert with args [ -l 3 -a MQ -e A60107 -f gbahevpl41.krf.express.RFR -u "http://messagemonitor.krf.express.RFR/BMTM/alerts/BMMEventOutput.1508657897.215.812.html" -m "gbahevpl41.krf.express.RFR:QPAHECL1:QTW1.BOQ.109 ==> eRIC for MWW not processed messages for 5 minutes"]
BmmRaiseAlert.sh::369: Completed successfully

but i need to read each file and save with file name front of the output..

Example

BmmRaiseAlert.sh.201710220838.6166.logfile.log

BmmRaiseAlert.sh::365: Calling appalert with args [ -l 3 -a MQ -e A60107 -f gbahevpl41.krf.express.RFR -u "http://messagemonitor.krf.express.RFR/BMTM/alerts/BMMEventOutput.1508657897.215.812.html" -m "gbahevpl41.krf.express.RFR:QPAHECL1:QTW1.BOQ.109 ==> eRIC for MWW not processed messages for 5 minutes"]
BmmRaiseAlert.sh::369: Completed successfully



BmmRaiseAlert.sh.201710220838.6177.logfile.log

BmmRaiseAlert.sh::365: Calling appalert with args [ -l 3 -a MQ -e A60107 -f gbahevpl41.krf.express.RFR -u "http://messagemonitor.krf.express.RFR/BMTM/alerts/BMMEventOutput.1508657897.215.812.html" -m "gbahevpl41.krf.express.RFR:QPAHECL1:QTW1.BOQ.109 ==> eRIC for MWW not processed messages for 5 minutes"]
BmmRaiseAlert.sh::369: Completed successfully

RudiC · October 24, 2017, 4:55am

That spec still is quite confusing. How far would

grep -Ei 'Calling appalert|AlertConfigResult|Completed succ' *.log

get you?

darling · October 24, 2017, 5:49am

Yes, it worked RudiC

MadeInGermany · October 24, 2017, 7:06am

As Don stated already, the -i option makes things slower, and is not required here.
The problem was that the (useless) use of cat the file names disappear:

cat *.log | egrep 'Calling appalert|Completed'

because egrep sees only one input stream.
Simply let egrep open the files, so it sees the file names and even prefixes them by default.

egrep 'Calling appalert|Completed' *.log

If you want to ensure it prefixes the file name even if there is only one log file, then do as Don suggested

egrep -H 'Calling appalert|Completed' *.log

or, if your grep -E or egrep does not support the -H option, use the /dev/null trick

egrep 'Calling appalert|Completed' /dev/null *.log

The trick is that grep sees one more file /dev/null that is empty (so will never match anything in it).