# Finding most common substrings

**URL:** <https://community.unix.com/t/finding-most-common-substrings/340013>\
**Category:** Shell Programming and Scripting\
**Created:** [February 22, 2014, 8:24pm UTC](https://community.unix.com/t/finding-most-common-substrings/340013 "2014-02-22T20:24:22Z")\
**Posts on this page:** 1\
**Showing post:** 4

<div class="post-metadata">

**Author:** ![Don\_Cragun](https://community.unix.com/user_avatar/community.unix.com/don_cragun/32/2682_2.png) [@Don\_Cragun](https://community.unix.com/u/Don_Cragun)\
**Post date:** [February 23, 2014, 12:05am UTC](https://community.unix.com/t/finding-most-common-substrings/340013/4 "2014-02-23T00:05:14Z")

</div>

With the simple awk program:

```nohighlight
#!/bin/ksh
awk '
NR > 1 {for(i = length($2) - 5; i >= 1; i--)
                c[substr($2, i, 6)]++
}
END { for(i in c)
                print c, i | "sort -rn"
}' file

```

and with `file` containing the data you provided in message #1, the output produced is:

```nohighlight
13 byebye
8 yebyeb
8 ebyeby
5 ypehel
5 typehe
5 pehell
5 llobye
5 hellob
5 elloby
5 ehello
5 catcat
4 tcatdo
4 ogcatc
4 gcatca
4 dogcat
4 catdog
4 atcatd
3 yedogc
3 ogbyeb
3 obyedo
3 lobyed
3 gbyeby
3 edogca
3 dogbye
3 byedog
2 tdogby
2 obyeby
2 lobyeb
2 atdogb
1 ttypeh
1 tdogdo
1 tcatty
1 ogdogb
1 gdogby
1 dogdog
1 cattyp
1 attype
1 atdogd
1 atcatt

```

so "byebye" is indeed the most common six character substring in the 2nd column of your input. But "catdog" and "typehe" don't even come close. To just print the most commonly occurring substring or (if more than one substring appears the same number of times as the first most common substring) substrings, try:

```nohighlight
#!/bin/ksh
awk '
NR > 1 {for(i = length($2) - 5; i >= 1; i--)
                c[substr($2, i, 6)]++
}
END { for(i in c)
                print c, i | "sort -rn"
}' file | awk '
NR == 1 {c = $1}
$1 == c {print $2}'

```

which just prints:

```nohighlight
byebye

```

If you want to try this on a Solaris/SunOS system, change `awk` from the default `/usr/bin/awk` to `/usr/xpg4/bin/awk` , `/usr/xpg6/bin/awk` , or `nawk` .

I use the Korn shell, but any shell that supports basic Bourne shell or POSIX standard shell syntax can be used instead.

---

_[View the full topic](https://community.unix.com/t/finding-most-common-substrings/340013)._