# Better way to run this perl command **URL:** \ **Category:** Shell Programming and Scripting\ **Created:** [May 30, 2012, 1:16pm UTC](https://community.unix.com/t/better-way-to-run-this-perl-command/311064 "2012-05-30T13:16:19Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![SkySmart](https://community.unix.com/letter_avatar/skysmart/32/5_5575768a8748004e209b776fc1b2916d.png) [@SkySmart](https://community.unix.com/u/SkySmart)\ **Post date:** [May 30, 2012, 1:16pm UTC](https://community.unix.com/t/better-way-to-run-this-perl-command/311064/1 "2012-05-30T13:16:19Z")
i'm working with files that are huge in size. over 3GB. and i need to do a lot of pattern matching. I need a way to grep for what i want, using a tool that is available across most unix systems. i initially was gungho about grep, but not all capablities of grep are available on all OSes. so im focusing on a command like this. how can i make it run faster? ```nohighlight date ; perl -ne 'print if /core.manager.referral/' $DATAFILE | perl -ne 'print if /ERROR getting Referral/' | wc -l ; date ``` this code searches for all lines that contain both "core.manager.referral " AND "ERROR getting Referral" shell = bash os = linux red hat, sunos datafile = 4GB ---
**Author:** ![joeyg](https://community.unix.com/user_avatar/community.unix.com/joeyg/32/780_2.png) [@joeyg](https://community.unix.com/u/joeyg)\ **Post date:** [May 30, 2012, 1:37pm UTC](https://community.unix.com/t/better-way-to-run-this-perl-command/311064/2 "2012-05-30T13:37:19Z")
Seems all you need is plain-vanilla grep, not any fancy options. Grep on what you think will have the fewest matches, to make a much smaller file. Then grep on the other option. Or, are you trying to determine if perl & if is faster than grep? ---
**Author:** ![pravin27](https://community.unix.com/letter_avatar/pravin27/32/5_5575768a8748004e209b776fc1b2916d.png) [@pravin27](https://community.unix.com/u/pravin27)\ **Post date:** [May 30, 2012, 1:38pm UTC](https://community.unix.com/t/better-way-to-run-this-perl-command/311064/3 "2012-05-30T13:38:05Z")
It would be good if you can post some small part of your input file ---
**Author:** ![SkySmart](https://community.unix.com/letter_avatar/skysmart/32/5_5575768a8748004e209b776fc1b2916d.png) [@SkySmart](https://community.unix.com/u/SkySmart)\ **Post date:** [May 30, 2012, 1:43pm UTC](https://community.unix.com/t/better-way-to-run-this-perl-command/311064/4 "2012-05-30T13:43:30Z")
> [@joeyg](#): > > Seems all you need is plain-vanilla grep, not any fancy options. > Grep on what you think will have the fewest matches, to make a much smaller file. Then grep on the other option. > > Or, are you trying to determine if perl & if is faster than grep? yes i'm trying to see if perl is faster than grep. and posting the input file is not an option at all. and the strings i picked out to search for necessary. i deliberately picked them. ---
**Author:** ![Corona688](https://community.unix.com/letter_avatar/corona688/32/5_5575768a8748004e209b776fc1b2916d.png) [@Corona688](https://community.unix.com/u/Corona688)\ **Post date:** [May 30, 2012, 1:47pm UTC](https://community.unix.com/t/better-way-to-run-this-perl-command/311064/5 "2012-05-30T13:47:08Z")
perl is massively huger than grep, that's a good sign it's not going to be faster. Running it twice, that's certainly not going to be faster. Piping it into wc -l is going to be slower yet. How about awk? Getting rid of pipes and external utilities is a good way to speed up a program, and awk can easily do everything you want here in one program with no pipes or external utilities (besides awk itself of course). ```nohighlight awk '/core.manager.referral/ && /ERROR getting Referral/ { N++} END { print N }' filename ``` ---
**Author:** ![SkySmart](https://community.unix.com/letter_avatar/skysmart/32/5_5575768a8748004e209b776fc1b2916d.png) [@SkySmart](https://community.unix.com/u/SkySmart)\ **Post date:** [May 30, 2012, 1:51pm UTC](https://community.unix.com/t/better-way-to-run-this-perl-command/311064/6 "2012-05-30T13:51:16Z")
> [@corona688](#): > > perl is massively huger than grep, that's a good sign it's not going to be faster. Running it twice, that's certainly not going to be faster. Piping it into wc -l is going to be slower yet. > > How about awk? Getting rid of pipes and external utilities is a good way to speed up a program, and awk can easily do everything you want here in one program with no pipes or external utilities (besides awk itself of course). > > ```plaintext > awk '/core.manager.referral/ && /ERROR getting Referral/ { N++} END { print N }' filename > > ``` thank you. i tried this, and the command is still slow to finish. im open to any one-liner type of command like this that will be the fastest. ---
**Author:** ![Corona688](https://community.unix.com/letter_avatar/corona688/32/5_5575768a8748004e209b776fc1b2916d.png) [@Corona688](https://community.unix.com/u/Corona688)\ **Post date:** [May 30, 2012, 1:53pm UTC](https://community.unix.com/t/better-way-to-run-this-perl-command/311064/7 "2012-05-30T13:53:57Z")
> [@skysmart](#): > > thank you. i tried this, and the command is still slow to finish. You are working with FOUR GIGABYTES of data. Of course it's slow. What's the file size, divided by your maximum disk speed? How much RAM do you have for cache? What's the bus speed of your RAM? I don't think you're going to improve much on the awk version. You might try tricks with index() instead of regexes. ---
**Author:** ![SkySmart](https://community.unix.com/letter_avatar/skysmart/32/5_5575768a8748004e209b776fc1b2916d.png) [@SkySmart](https://community.unix.com/u/SkySmart)\ **Post date:** [May 30, 2012, 1:59pm UTC](https://community.unix.com/t/better-way-to-run-this-perl-command/311064/8 "2012-05-30T13:59:58Z")
> [@corona688](#): > > You are working with FOUR GIGABYTES of data. Of course it's slow. What's the file size, divided by your maximum disk speed? How much RAM do you have for cache? What's the bus speed of your RAM? > > I don't think you're going to improve much on the awk version. You might try tricks with index() instead of regexes. can you please elaborate on the index()? examples? ---
**Author:** ![Corona688](https://community.unix.com/letter_avatar/corona688/32/5_5575768a8748004e209b776fc1b2916d.png) [@Corona688](https://community.unix.com/u/Corona688)\ **Post date:** [May 30, 2012, 2:07pm UTC](https://community.unix.com/t/better-way-to-run-this-perl-command/311064/9 "2012-05-30T14:07:48Z")
```nohighlight awk 'index($0, "not a regex") && index($0, "also not a regex") { N++ } END { print N }' ``` ...but I'm certain it's not going to help, because my questions weren't rhetorical. CPU probably isn't your bottleneck, and you're only going to get minor speed improvements from changing the program. How fast is your disk? How fast is your memory? How large is your memory? These are important questions when throwing around huge files. ---
**Author:** ![SkySmart](https://community.unix.com/letter_avatar/skysmart/32/5_5575768a8748004e209b776fc1b2916d.png) [@SkySmart](https://community.unix.com/u/SkySmart)\ **Post date:** [May 30, 2012, 2:14pm UTC](https://community.unix.com/t/better-way-to-run-this-perl-command/311064/10 "2012-05-30T14:14:15Z")
> [@corona688](#): > > ```plaintext > awk 'index($0, "not a regex") && index($0, "also not a regex") { N++ } END { print N }' > > ``` > > ...but I'm certain it's not going to help, because my questions weren't rhetorical. CPU probably isn't your bottleneck, and you're only going to get minor speed improvements from changing the program. > > How fast is your disk? How fast is your memory? How large is your memory? > > These are important questions when throwing around huge files. yeah you're right. i have to do this on hundreds of servers that have different hardware so i cant really specify. thank you ---
**Author:** ![Corona688](https://community.unix.com/letter_avatar/corona688/32/5_5575768a8748004e209b776fc1b2916d.png) [@Corona688](https://community.unix.com/u/Corona688)\ **Post date:** [May 30, 2012, 2:29pm UTC](https://community.unix.com/t/better-way-to-run-this-perl-command/311064/11 "2012-05-30T14:29:51Z")
With this much data to go through, it may be better to handle the data as-it-happens instead of after the fact. Or cache the results for later if you can't do that, since it's just not going to be fast, period.