Searching mutiple word - Tuning grep command

UNIX for Beginners Q & A UNIX for Dummies Questions & Answers
1

Hi all,

I have a log file which is more than 1GB, i need to take count from the log file for two strings. i am using the below command but it take a long time to excetue, i need to tune this. Please help me

cat /logs/gcbs/gcbsTrace.log | grep -i "ViewStatementBusinessLogic" | grep -c -i "MYCACommunicationException"

similary i need to check the count for other execption too. Is there any simple way is avialable.

Regards,
Senthilkumar AK

2

It's probably better if you post realistic examples and keep all the discussion in a single thread. See Please suggest some changes in my code and Guidance needed for quick script

As such, I would recommend that you combine all your grep statements to a single sed script.

sed -n '/regex1/s/.*\(regex2\).*/\1/p
/regex3/s/.*\(regex4\).*/p
/regex5/s/.*\(regex6\).*/p' /logs/gcbs/gcbsTrace.log | sort | uniq -c

That way, you only scan the whole file once, and spend less time waiting for the disk.

3

Thanks era,

The Sed command also takes the same time as the grep takes, also i take the suggestion which you posted and continue in this thread alone.

4

If you really need to improve performance, you need to move away from a shell script
and write something in C or C++

5

ok is it i can write a c program and use this in Linux machine..? I searched in the machine and found perl is there.

6

The point with the sed is that it takes more time than a single grep, but you can replace all the greps with a single sed script.

Unless you are a skilled C programmer, I doubt that you can make this a lot faster by switching languages. C programs are compiled, you don't need the C compiler on the box itself in order to run the resulting binary.

7

You can give awk a try:

awk '{$0=tolower($0)}
/viewstatementbusinesslogic/ && /mycacommunicationexception/{c++}END{print c++}
' file

Regards

8

Hi senthilkumar_ak,

You can use Perl. It will give you better results than grep i think.
Here are few tips for searching ...

  1. do not store the file contents in an array, if the file size is huge
  2. loop thru using the file handle ex: while (my $line = <FILE>) {
  3. to make things faster, use variables as counters. If you want to match say 5 times ... you need not have to parse the entire file in loop, exit after 5 matches using a counter var.

Regards,
Visit my Perl blog for tips on Perl,
TechDiary

9

Thanks vikas.. I am beginner in perl i will use your tips, may be i will come up with some query to you if i encountered any errors.