# awk command optimization

**URL:** <https://community.unix.com/t/awk-command-optimization/346282>\
**Category:** Shell Programming and Scripting\
**Created:** [September 13, 2014, 10:04am UTC](https://community.unix.com/t/awk-command-optimization/346282 "2014-09-13T10:04:32Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![SkySmart](https://community.unix.com/letter_avatar/skysmart/32/5_5575768a8748004e209b776fc1b2916d.png) [@SkySmart](https://community.unix.com/u/SkySmart)\
**Post date:** [September 13, 2014, 10:04am UTC](https://community.unix.com/t/awk-command-optimization/346282/1 "2014-09-13T10:04:32Z")

</div>

```nohighlight
gawk -v sw="error|fail|panic|accepted" 'NR>1 && NR <=128500 {
                                                                                for (w in a)
                                                                                {
                                                                                        if ($0 ~ a[w])
                                                                                                d[a[w]]++
                                                                                }
                                                                }
                                                                BEGIN {
                                                                        c = split(sw,a,"[|]")
                                                                }
                                                                END {
                                                                for (i in a)
                                                                {
                                                                        o = o (a"="(d[a]?d[a]:0)",")
                                                                }
                                                                        sub(",*$","",o)
                                                                        print o
                                                                }' /var/log/treg.test

```

the above code works majestically when searching for multiple strings in a log.

the problem is, as the log gets bigger (i.e. 5MB), the time it takes to search for all the strings gets longer as well. took 2 seconds to search a 5MB file using this code. had the file been bigger, say 10MB, it would take longer.

so i'm wondering, can this code be optimized at all to make it run faster? maybe if the strings were read from a separate file it would help speed things up??

code runs on linux redhat / ubuntu platforms

---

<div class="post-metadata">

**Author:** ![jim\_mcnamara](https://community.unix.com/letter_avatar/jim_mcnamara/32/5_5575768a8748004e209b776fc1b2916d.png) [@jim\_mcnamara](https://community.unix.com/u/jim_mcnamara)\
**Post date:** [September 13, 2014, 10:48am UTC](https://community.unix.com/t/awk-command-optimization/346282/2 "2014-09-13T10:48:53Z")

</div>

FWIW -

```nohighlight
egrep -c '(error|fail|panic|accepted)' logfile

```

Does a lot of what your awk code does, not all of it.

```nohighlight
for (w in a)
 {
         if ($0 ~ a[w])
                 d[a[w]]++
 }

```

This code above means you loop 5 times on every line. I do not think regex in awk supports alternation, someone who knows more please comment. But that would be the first place to attack your problem. And if you search for more terms your program will iterate more times over each line of input.

This is the same problem we have when we use `grep -f list_of_items filename` with a large number of entries in `list_of_items` .

Edit: the red comment is flat wrong. Alternation is possible. You can rewrite the main loop to use it.

```nohighlight
{ /error|wanting|panic|failure/ } { [define array here]++..... }

```

---

<div class="post-metadata">

**Author:** ![shamrock](https://community.unix.com/letter_avatar/shamrock/32/5_5575768a8748004e209b776fc1b2916d.png) [@shamrock](https://community.unix.com/u/shamrock)\
**Post date:** [September 13, 2014, 1:09pm UTC](https://community.unix.com/t/awk-command-optimization/346282/3 "2014-09-13T13:09:27Z")

</div>

Can you post a sample of the input file as well...

---

<div class="post-metadata">

**Author:** ![SkySmart](https://community.unix.com/letter_avatar/skysmart/32/5_5575768a8748004e209b776fc1b2916d.png) [@SkySmart](https://community.unix.com/u/SkySmart)\
**Post date:** [September 13, 2014, 2:02pm UTC](https://community.unix.com/t/awk-command-optimization/346282/4 "2014-09-13T14:02:07Z")

</div>

> [@shamrock](#):
>
> Can you post a sample of the input file as well...

the input file is just any data file. no set format. the code is used to list the number of specific patterns in a file. so the file does not matter.

---

<div class="post-metadata">

**Author:** ![disedorgue](https://community.unix.com/user_avatar/community.unix.com/disedorgue/32/22261_2.png) [@disedorgue](https://community.unix.com/u/disedorgue)\
**Post date:** [September 13, 2014, 2:21pm UTC](https://community.unix.com/t/awk-command-optimization/346282/5 "2014-09-13T14:21:39Z")

</div>

Hi,  
You can try:

```nohighlight
gawk -v sw="error|fail|panic|accepted" 'NR>1 && NR <=128500 && match($0,"/"sw"/") {
											d[substr($0,RSTART,RLENGTH)]++
                                                                                }

                                                                BEGIN {
                                                                        c = split(sw,a,"[|]")
                                                                }
                                                                END {
                                                                for (i in a)
                                                                {
                                                                        o = o (a"="(d[a]?d[a]:0)",")
                                                                }
                                                                        sub(",*$","",o)
                                                                        print o
                                                                }' /var/log/treg.test

```

Regards.

---

<div class="post-metadata">

**Author:** ![shamrock](https://community.unix.com/letter_avatar/shamrock/32/5_5575768a8748004e209b776fc1b2916d.png) [@shamrock](https://community.unix.com/u/shamrock)\
**Post date:** [September 13, 2014, 2:23pm UTC](https://community.unix.com/t/awk-command-optimization/346282/6 "2014-09-13T14:23:29Z")

</div>

Well you could give this [g]awk a try...

```nohighlight
gawk '{
    for (i=1; i<=NF; i++)
        if ($i ~ "^(error|fail|panic|accepted)$")
            a[$i]++
} END {
    for (i in a) {
        n++
        printf("%s=%s%s", i, a, (n < 4 ? ", " : "\n"))
    }
}' file

```

---

<div class="post-metadata">

**Author:** ![SkySmart](https://community.unix.com/letter_avatar/skysmart/32/5_5575768a8748004e209b776fc1b2916d.png) [@SkySmart](https://community.unix.com/u/SkySmart)\
**Post date:** [September 13, 2014, 3:01pm UTC](https://community.unix.com/t/awk-command-optimization/346282/7 "2014-09-13T15:01:37Z")

</div>

> [@disedorgue](#):
>
> Hi,  
> You can try:
> 
> ```plaintext
> gawk -v sw="error|fail|panic|accepted" 'NR>1 && NR <=128500 && match($0,"/"sw"/") {
> d[substr($0,RSTART,RLENGTH)]++
> }
> 
> BEGIN {
> c = split(sw,a,"[|]")
> }
> END {
> for (i in a)
> {
> o = o (a"="(d[a]?d[a]:0)",")
> }
> sub(",*$","",o)
> print o
> }' /var/log/treg.test
> 
> ```
> 
> Regards.

thank you so much!

this looks promising. when i run it though, it only gives a count for one of the strings even though there are lines in the data file that contain the other strings:

```plaintext
accepted=0,error=0,fail=3859,panic=0

```

---------- Post updated at 02:01 PM ---------- Previous update was at 01:58 PM ----------

> [@shamrock](#):
>
> Well you could give this [g]awk a try...
> 
> ```plaintext
> gawk '{
> for (i=1; i<=NF; i++)
> if ($i ~ "^(error|fail|panic|accepted)$")
> a[$i]++
> } END {
> for (i in a) {
> n++
> printf("%s=%s%s", i, a, (n < 4 ? ", " : "\n"))
> }
> }' file
> 
> ```

this looks quite promising as well. thank you so much!!!

looks like the code is written in such a way that it only counts the number of lines that contain just the specific patterns specified. but i believe i can play with it some more.

one question. is the "n \< 4" setting a limit of patterns that can be specified?

btw, this completed in under .3 seconds on a 5mb file. so very good news!!!

---

<div class="post-metadata">

**Author:** ![disedorgue](https://community.unix.com/user_avatar/community.unix.com/disedorgue/32/22261_2.png) [@disedorgue](https://community.unix.com/u/disedorgue)\
**Post date:** [September 13, 2014, 3:23pm UTC](https://community.unix.com/t/awk-command-optimization/346282/8 "2014-09-13T15:23:34Z")

</div>

Strange, this awk script work fine for me.  
The case where not work is when a line with more one pattern (only the first pattern find).  
In your data file, have you more that one pattern by line ?

---

<div class="post-metadata">

**Author:** ![Don\_Cragun](https://community.unix.com/user_avatar/community.unix.com/don_cragun/32/2682_2.png) [@Don\_Cragun](https://community.unix.com/u/Don_Cragun)\
**Post date:** [September 13, 2014, 3:38pm UTC](https://community.unix.com/t/awk-command-optimization/346282/9 "2014-09-13T15:38:34Z")

</div>

> [@skysmart](#):
>
> ```plaintext
> gawk -v sw="error|fail|panic|accepted" 'NR>1 && NR <=128500 {
> for (w in a)
> {
> if ($0 ~ a[w])
> d[a[w]]++
> }
> }
> BEGIN {
> c = split(sw,a,"[|]")
> }
> END {
> for (i in a)
> {
> o = o (a"="(d[a]?d[a]:0)",")
> }
> sub(",*$","",o)
> print o
> }' /var/log/treg.test
> 
> ```
> 
> the above code works majestically when searching for multiple strings in a log.
> 
> the problem is, as the log gets bigger (i.e. 5MB), the time it takes to search for all the strings gets longer as well. took 2 seconds to search a 5MB file using this code. had the file been bigger, say 10MB, it would take longer.
> 
> so i'm wondering, can this code be optimized at all to make it run faster? maybe if the strings were read from a separate file it would help speed things up??
> 
> code runs on linux redhat / ubuntu platforms

No, reading your patterns from a file will not be faster than using `split()` to extract them from a string!

What is on line 1 and on lines 128501 to the end of your input file. If you can adjust your counters for what I assume is constant data on those lines you can skip two tests that are being performed on every line. (The tests may be fast individually, but performing more than a quarter of a million fast tests adds up.)

If no more than one or your search patterns could appear on a single input line , changing:

```plaintext
for (w in a) {
        if ($0 ~ a[w])
                d[a[w]]++
}

```

to:

```plaintext
for (w in a) {
        if ($0 ~ a[w]) {
                d[a[w]]++
                next
        }
}

```

would speed things up.

Depending on what percentage of your input lines contain one or more of the search patterns, adding the test:

```plaintext
if($0 ~ sw) {
        above for loop
}

```

may speed things up or slow things down. If one of the search patterns appears on every input line, it will slow things down. If none of the search patterns appear on a vast majority of your input lines, it will speed things up. Your mileage will vary depending on your version of `awk` and your input data.

---

<div class="post-metadata">

**Author:** ![SkySmart](https://community.unix.com/letter_avatar/skysmart/32/5_5575768a8748004e209b776fc1b2916d.png) [@SkySmart](https://community.unix.com/u/SkySmart)\
**Post date:** [September 13, 2014, 6:54pm UTC](https://community.unix.com/t/awk-command-optimization/346282/10 "2014-09-13T18:54:48Z")

</div>

> [@shamrock](#):
>
> Well you could give this [g]awk a try...
> 
> ```plaintext
> gawk '{
> for (i=1; i<=NF; i++)
> if ($i ~ "^(error|fail|panic|accepted)$")
> a[$i]++
> } END {
> for (i in a) {
> n++
> printf("%s=%s%s", i, a, (n < 4 ? ", " : "\n"))
> }
> }' file
> 
> ```

so this code seems to be doing what i need. however, it doesn't appear to be finding strings that have spaces in them. here's how i'm running it:

```plaintext
gawk '{
    for (i=1; i<=NF; i++)
        if ($i ~ "(error|fail|panic|open database|accepted)")
            a[$i]++
} END {
    for (i in a) {
        n++
        printf("%s=%s%s", i, a, (n < 4 ? ", " : "\n"))
    }
}' file

```

it finds every other string except the "open database" one. i tried replacing it with "open.\*database" and that still didn't work. can this be tweaked to accept strings with spaces?

i also tried:

```plaintext
gawk '{
    for (i=1; i<=NF; i++)
        if ($i ~ "/error|fail|panic|open database|accepted/")
            a[$i]++
} END {
    for (i in a) {
        n++
        printf("%s=%s%s", i, a, (n < 4 ? ", " : "\n"))
    }
}' file

```

---

<div class="post-metadata">

**Author:** ![Don\_Cragun](https://community.unix.com/user_avatar/community.unix.com/don_cragun/32/2682_2.png) [@Don\_Cragun](https://community.unix.com/u/Don_Cragun)\
**Post date:** [September 13, 2014, 10:01pm UTC](https://community.unix.com/t/awk-command-optimization/346282/11 "2014-09-13T22:01:38Z")

</div>

> [@skysmart](#):
>
> so this code seems to be doing what i need. however, it doesn't appear to be finding strings that have spaces in them. here's how i'm running it:
> 
> ```plaintext
> 
> gawk '{
> for (i=1; i<=NF; i++)
> if ($i ~ "(error|fail|panic|open database|accepted)")
> a[$i]++
> } END {
> for (i in a) {
> n++
> printf("%s=%s%s", i, a, (n < 4 ? ", " : "\n"))
> }
> }' file
> 
> ```
> 
> it finds every other string except the "open database" one. i tried replacing it with "open.\*database" and that still didn't work. can this be tweaked to accept strings with spaces?  
> ... ... ...

Not easily. When your field separator is whitespace characters (the `awk` default), trying to match a single field that contains one of your field separators is always going to fail. Furthermore, this code will combine your desired patterns with surrounding non-space characters. For instance, with the following contents in `file` :

```plaintext
error open
open database
database fail
fail accepted
		if ($i ~ "(error|fail|panic|open database|accepted)")
accepted error
error, fail, panic, open database, accepted

```

the code:

```plaintext
gawk '{
	for (i=1; i<=NF; i++)
		if ($i ~ "error|fail|panic|open database|accepted")
			a[$i]++
} END {
	for (i in a) {
		n++
		printf("%s=%s%s", i, a, (n < 4 ? ", " : "\n"))
	}
}' file

```

produces the output:

```plaintext
error=2, fail=2, fail,=1, error,=1
accepted.=1
accepted=2
panic,=1
"(error|fail|panic|open=1
database|accepted)")=1

```

Note that the 4 shown in red above controls how many patterns are shown on the 1st line of the output. If all of your patterns only appeared as complete fields (with no punctuation and no additional letter [such as errors or panics]), you would want that number to be the same as the number of patterns you have in your `|` separated list of patterns.

---

<div class="post-metadata">

**Author:** ![shamrock](https://community.unix.com/letter_avatar/shamrock/32/5_5575768a8748004e209b776fc1b2916d.png) [@shamrock](https://community.unix.com/u/shamrock)\
**Post date:** [September 14, 2014, 1:59pm UTC](https://community.unix.com/t/awk-command-optimization/346282/12 "2014-09-14T13:59:53Z")

</div>

> [@skysmart](#):
>
> one question. is the "n \< 4" setting a limit of patterns that can be specified?

That was specific to your question as you were looking for 4 patterns and if you want something generic then take out the n...

---------- Post updated at 12:59 PM ---------- Previous update was at 12:47 PM ----------

> [@skysmart](#):
>
> so this code seems to be doing what i need. however, it doesn't appear to be finding strings that have spaces in them. here's how i'm running it:
> 
> ```plaintext
> 
> gawk '{
> for (i=1; i<=NF; i++)
> if ($i ~ "(error|fail|panic|open database|accepted)")
> a[$i]++
> } END {
> for (i in a) {
> n++
> printf("%s=%s%s", i, a, (n < 4 ? ", " : "\n"))
> }
> }' file
> 
> ```
> 
> it finds every other string except the "open database" one. i tried replacing it with "open.\*database" and that still didn't work. can this be tweaked to accept strings with spaces?

The gawk above wont work as it splits each line into fields delimited by whitespace...and to find strings or phrases with whitespace you'd have to do something like...

```plaintext
gawk '{
    a["error"] += gsub("(^| )error( |$)", "&")
    a["fail"] += gsub("(^| )fail( |$)", "&")
    a["panic"] += gsub("(^| )panic( |$)", "&")
    a["accepted"] += gsub("(^| )accepted( |$)", "&")
    a["open database"] += gsub("(^| )open database( |$)", "&")
} END {
    for (i in a)
        printf("%s=%s\n", i, a)
}' treg.test

```

Hope this helps...

---

<div class="post-metadata">

**Author:** ![SkySmart](https://community.unix.com/letter_avatar/skysmart/32/5_5575768a8748004e209b776fc1b2916d.png) [@SkySmart](https://community.unix.com/u/SkySmart)\
**Post date:** [September 14, 2014, 2:59pm UTC](https://community.unix.com/t/awk-command-optimization/346282/13 "2014-09-14T14:59:01Z")

</div>

> [@shamrock](#):
>
> That was specific to your question as you were looking for 4 patterns and if you want something generic then take out the n...
> 
> ---------- Post updated at 12:59 PM ---------- Previous update was at 12:47 PM ----------
> 
> The gawk above wont work as it splits each line into fields delimited by whitespace...and to find strings or phrases with whitespace you'd have to do something like...
> 
> ```plaintext
> gawk '{
> a["error"] += gsub("(^| )error( |$)", "&")
> a["fail"] += gsub("(^| )fail( |$)", "&")
> a["panic"] += gsub("(^| )panic( |$)", "&")
> a["accepted"] += gsub("(^| )accepted( |$)", "&")
> a["open database"] += gsub("(^| )open database( |$)", "&")
> } END {
> for (i in a)
> printf("%s=%s\n", i, a)
> }' treg.test
> 
> ```
> 
> Hope this helps...

Yes, it helps a lot!!!

can i pass the strings as an argument instead of hardcoding it in?

```plaintext
strvar="error|fail|panic|open database|accepted"

```

---

<div class="post-metadata">

**Author:** ![Chubler\_XL](https://community.unix.com/user_avatar/community.unix.com/chubler_xl/32/2077_2.png) [@Chubler\_XL](https://community.unix.com/u/Chubler_XL)\
**Post date:** [September 14, 2014, 7:21pm UTC](https://community.unix.com/t/awk-command-optimization/346282/14 "2014-09-14T19:21:24Z")

</div>

I would be interested to see how the performance of these solutions compare with something like:

```nohighlight
egrep -ow "(error|fail|panic|open database|accepted)" treg.test | sort | uniq -c

```

---

<div class="post-metadata">

**Author:** ![Don\_Cragun](https://community.unix.com/user_avatar/community.unix.com/don_cragun/32/2682_2.png) [@Don\_Cragun](https://community.unix.com/u/Don_Cragun)\
**Post date:** [September 14, 2014, 10:14pm UTC](https://community.unix.com/t/awk-command-optimization/346282/15 "2014-09-14T22:14:53Z")

</div>

SkySmart,  
I have yet to see a clear definition of what the submitter wants to be matched by the patterns given. For example, with the following input:

```nohighlight
error open
open database
database fail
fail accepted
		if ($i ~ "(error|fail|panic|open database|accepted)")
accepted error

error, fail, panic, open database, accepted.
error123 error456:error789
error error error

```

should `error` match `error|` , `error,` , and `error123` as well as when `error` is at the start or end of a line and when it is preceded and followed by whitespace characters. The following seems to do what is wanted if all of the above are supposed to match:

```nohighlight
strvar="error|fail|panic|open database|accepted"
awk -v patlist="$strvar" '
BEGIN {	npat = split(patlist, pl, "[|]")
	for(i = 1; i <= npat; i++)
		pat = "(^|[^[:alpha:]])" pl "([^[:alpha:]]|$)"
		# for(i=1; i <= npat; i++)
		#	printf("pl[%d]=%s, pat[%d]=%s\n", i, pl, i, pat)
}
$0 ~ patlist {	# printf("NR=%d, $0=%s\n", NR, $0)
	for(i = 1; i <= npat; i++) {
		a += gsub(pat, "&")
		# printf("a[%s] = %d (for %s)\n", i, a, pl)
	}
}
END {	for(i = 1; i <= npat; i++)
		printf("%s=%s\n", pl, a)
}' file

```

If you don't want to match `123error` and `error123` , change both occurrences of `:alpha:` in the code above to `:alnum:` .

The the vast majority of your input lines will contain one or more of your search patterns, this will probably run faster if you remove the code shown in red.

With the above sample input file, this script produces the output:

```nohighlight
error=9
fail=4
panic=2
open database=3
accepted=4

```

You also have yet to explain why your script was eliminating line 1 and lines 128501 and above from your input file. Until we know what is special about these lines and what is on them, we won't be able to help you make adjustments to these `awk` and `egrep` commands to solve your problem. I agree with Chubler\_XL that `egrep` looks like a better solution than `awk` , but depending on what you're trying to match, the ERE operand may need to be significantly modified and the -w option removed before invoking `egrep` . Furthermore, if some lines do need to be treated specially, `awk` becomes more attractive.

---

<div class="post-metadata">

**Author:** ![SkySmart](https://community.unix.com/letter_avatar/skysmart/32/5_5575768a8748004e209b776fc1b2916d.png) [@SkySmart](https://community.unix.com/u/SkySmart)\
**Post date:** [September 14, 2014, 11:17pm UTC](https://community.unix.com/t/awk-command-optimization/346282/16 "2014-09-14T23:17:57Z")

</div>

thanks everyone. this has now been resolved. i took bits and pieces from every single post in here to get what i want.

thank you guys!