Better way to run this perl command

SkySmart · May 30, 2012, 1:16pm

i'm working with files that are huge in size. over 3GB. and i need to do a lot of pattern matching. I need a way to grep for what i want, using a tool that is available across most unix systems.

i initially was gungho about grep, but not all capablities of grep are available on all OSes.

so im focusing on a command like this. how can i make it run faster?

date ; perl -ne 'print if /core.manager.referral/' $DATAFILE | perl -ne 'print if /ERROR getting Referral/' | wc -l ; date

this code searches for all lines that contain both "core.manager.referral " AND "ERROR getting Referral"

shell = bash
os = linux red hat, sunos
datafile = 4GB

joeyg · May 30, 2012, 1:37pm

Seems all you need is plain-vanilla grep, not any fancy options.
Grep on what you think will have the fewest matches, to make a much smaller file. Then grep on the other option.

Or, are you trying to determine if perl & if is faster than grep?

pravin27 · May 30, 2012, 1:38pm

It would be good if you can post some small part of your input file

SkySmart · May 30, 2012, 1:43pm

yes i'm trying to see if perl is faster than grep.

and posting the input file is not an option at all. and the strings i picked out to search for necessary. i deliberately picked them.

Corona688 · May 30, 2012, 1:47pm

perl is massively huger than grep, that's a good sign it's not going to be faster. Running it twice, that's certainly not going to be faster. Piping it into wc -l is going to be slower yet.

How about awk? Getting rid of pipes and external utilities is a good way to speed up a program, and awk can easily do everything you want here in one program with no pipes or external utilities (besides awk itself of course).

awk '/core.manager.referral/ && /ERROR getting Referral/ { N++} END { print N }' filename

SkySmart · May 30, 2012, 1:51pm

corona688:

perl is massively huger than grep, that's a good sign it's not going to be faster. Running it twice, that's certainly not going to be faster. Piping it into wc -l is going to be slower yet.

How about awk? Getting rid of pipes and external utilities is a good way to speed up a program, and awk can easily do everything you want here in one program with no pipes or external utilities (besides awk itself of course).
awk '/core.manager.referral/ && /ERROR getting Referral/ { N++} END { print N }' filename

thank you. i tried this, and the command is still slow to finish.

im open to any one-liner type of command like this that will be the fastest.

Corona688 · May 30, 2012, 1:53pm

You are working with FOUR GIGABYTES of data. Of course it's slow. What's the file size, divided by your maximum disk speed? How much RAM do you have for cache? What's the bus speed of your RAM?

I don't think you're going to improve much on the awk version. You might try tricks with index() instead of regexes.

SkySmart · May 30, 2012, 1:59pm

can you please elaborate on the index()? examples?

Corona688 · May 30, 2012, 2:07pm

awk 'index($0, "not a regex") && index($0, "also not a regex") { N++ } END { print N }'

...but I'm certain it's not going to help, because my questions weren't rhetorical. CPU probably isn't your bottleneck, and you're only going to get minor speed improvements from changing the program.

How fast is your disk? How fast is your memory? How large is your memory?

These are important questions when throwing around huge files.

SkySmart · May 30, 2012, 2:14pm

corona688:

awk 'index($0, "not a regex") && index($0, "also not a regex") { N++ } END { print N }'
...but I'm certain it's not going to help, because my questions weren't rhetorical. CPU probably isn't your bottleneck, and you're only going to get minor speed improvements from changing the program.

How fast is your disk? How fast is your memory? How large is your memory?

These are important questions when throwing around huge files.

yeah you're right. i have to do this on hundreds of servers that have different hardware so i cant really specify.

thank you

Corona688 · May 30, 2012, 2:29pm

With this much data to go through, it may be better to handle the data as-it-happens instead of after the fact. Or cache the results for later if you can't do that, since it's just not going to be fast, period.