Egrep find word that occurs twice in a row

spo_2138 · October 17, 2017, 8:29pm

Hi there I am trying to figure out and understand what the syntax would be to egrep lines that have a word occur twice in a row. the two words obviously should have a space between them and also it has to be case sensitive which I believe grep is by deffault. the closest I have come is...

grep '/.*\|.*\|/'

although I am kind of unsure what exactly I am saying here. Any
help would be much appreciated. I am not looking for just an
answer but an understanding of what the answer is doing as well

cheers

spo

Yoda · October 17, 2017, 8:46pm

egrep "pattern.*pattern" filename

awk '{n=gsub("pattern","&")}n>1' filename

Scrutinizer · October 17, 2017, 11:22pm

Try using regular expression's back reference \<num> , that lets you repeat a parenthesized subexpression.

grep '\(pattern\).*\1'

Another thing that can be useful are word boundaries. Your grep version may support them. they are \b , or \< and \> . Have a look at regular expressions..

spo_2138 · October 19, 2017, 5:33pm

Hi I am still having trouble getting either of these to work and still do not quite understand I need it to only recognize repeated whole words and neither are working for me. I am using and would like to use([a-z]+) as part of the code. Thanks for any and all help!

Don_Cragun · October 19, 2017, 6:33pm

According to the standards, extended regular expressions do not have back-references; only basic regular expressions have back-references. Therefore, with a standards conforming version of egrep (which the standards specify as grep -E (not egrep ), it is almost impossible to find a variable string that appears twice on a line.

If you use grep instead of egrep (as Scrutinizer suggested in post #3), you can use it to print lines that have a string matching the basic regular expression (AKA BRE) pattern followed by a second occurrence of the same string.

The command:

grep '/.*|.*/'

and the command:

grep '/.*\|.*/'

will both print lines that contain a / immediately following by any string of 0 or more characters followed by a | followed by any string of 0 or more characters followed by a / (which does not seem to in any way match what you said you're looking for).

If you're looking for a string of one or more lower-case alphabetic characters (in a locale where the underlying codeset is a superset of ASCII) immediately followed by a by a duplicate of that same string (with nothing between them), you could get that using the grep command:

grep '\([a-z]+\)\1'

and if you wanted to find two adjacent words that appear at the start of a line or immediately follow a space and are followed by a space or the end of a line that occur next to each other separated by a single space, that would be something like:

grep -e '^\([a-z\) \1$' -e '^\([a-z\) \1 ' -e ' \([a-z\) \1 ' -e ' \([a-z\) \1$'

As noted by Scrutinizer in post #7, the above BREs are incorrect. The corrected form (assuming there is a single space character between words) is:

grep -e '^\([a-z][a-z]*\) \1$' -e '^\([a-z][a-z]*\) \1 ' -e ' \([a-z][a-z]*\) \1 ' -e ' \([a-z][a-z]*\) \1$'

In the above command the first BRE looks for two identical lower-case words alone on a line, the 2nd BRE looks for two identical words at the start of a line followed by one or more other words, the 3rd BRE looks for two identical words following one or more other words an followed by one or more other words, and the last BRE looks for two identical lower-case words at the end of a line following one or more other words.

Some versions of grep do not conform to the standards unless additional parameters are specified to force standards conformance. Without knowing what operating system you're using, we have no way of knowing if this problem might affect you.

jgt · October 19, 2017, 9:23pm

don cragun:

According to the standards, extended regular expressions do not have back-references; only basic regular expressions have back-references. Therefore, with a standards conforming version of egrep (which the standards specify as grep -E (not egrep ), it is almost impossible to find a variable string that appears twice on a line.

If you use grep instead of egrep (as Scrutinizer suggested in post #3), you can use it to print lines that have a string matching the basic regular expression (AKA BRE) pattern followed by a second occurrence of the same string.

The command:
grep '/.*|.*/'
and the command:
grep '/.*\|.*/'
will both print lines that contain a / immediately following by any string of 0 or more characters followed by a | followed by any string of 0 or more characters followed by a / (which does not seem to in any way match what you said you're looking for).

If you're looking for a string of one or more lowe-case alphabetic characters (in a locale where the underlying codeset is a superset of ASCII) immediately followed by a by a duplicate of that same string (with nothing between them), you could get that using the grep command:
grep '$[a-z]+$\1'
and if you wanted to find two adjacent words that appear at the start of a line or immediately follow a space and are followed by a space or the end of a line that occur next to each other separated by a single space, that would be something like:
grep -e '^$[a-z$ \1$' -e '^$[a-z$ \1 ' -e ' $[a-z$ \1 ' -e ' $[a-z$ \1$'
In the above command the first BRE looks for two identical lower-case words alone on a line, the 2nd BRE looks for two identical words at the start of a line followed by one or more other words, the 3rd BRE looks for two identical words following one or more other words an followed by one or more other words, and the last BRE looks for two identical lower-case words at the end of a line following one or more other words.

Some versions of grep do not conform to the standards unless additional parameters are specified to force standards conformance. Without knowing what operating system you're using, we have no way of knowing if this problem might affect you.

you should be able to accomplish this by processing the file twice.

sed -e 's/abcde/xxxxx/' <inputfile|grep abcde |sed -e 's/xxxxx/abcde/'

Scrutinizer · October 19, 2017, 9:31pm

In addition to Don's suggestion :
grep does not know + , so you woud need to use \{1,\} instead.
In the example, a closing square bracket and repeat operators appears to be missing, so I think it would need to be modified like so:

grep -e '^\([a-z]\{1,\}\) \1$' -e '^\([a-z]\{1,\}\) \1 ' -e ' \([a-z]\{1,\}\) \1 ' -e ' \([a-z]\{1,\}\) \1$'

Where both the sub-pattern and its back reference are on word boundaries, either at the beginning followed by space, at the end preceded by space or in between space characters.

But without word boundary operators, it gets more complicated when the words do not have to be adjacent:

grep -e '^\([a-z]\{1,\}\) \([^ ]* \)*\1$' -e '^\([a-z]\{1,\}\) \([^ ]* \)*\1 ' -e ' \([a-z]\{1,\}\) \([^ ]* \)*\1 ' -e ' \([a-z]\{1,\}\) \([^ ]* \)*\1$'

Another thing to note that this is just the case where words are on the boundaries with a space. But there can be comma's, semicolons punctuations etcetera.

--
If you have GNU or BSD grep (as opposed to standard grep) then you can use word boundaries as an extension to regex, so it can be simplified into something like this:

grep '\<\([a-z]\{1,\}\)\>.*\<\1\>'

They also support back reference with extended regular expressions so, you can can do this:

grep -E '\<([a-z]+)\>.*\<\1\>'

Note in general instead of [a-z] , it is preferable to use [[:lower:]] for lowercase or [[:alpha:]] which matches both upper and lower case in all compliant code sets..

Don_Cragun · October 19, 2017, 10:23pm

scrutinizer:

In addition to Don's suggestion :
grep does not know + , so you woud need to use \{1,\} instead.
In the example, a closing square bracket and repeat operators appears to be missing, so I think it would need to be modified like so:
grep -e '^$[a-z]\{1,\}$ \1$' -e '^$[a-z]\{1,\}$ \1 ' -e ' $[a-z]\{1,\}$ \1 ' -e ' $[a-z]\{1,\}$ \1$'
Where both the sub-pattern and its back reference are on word boundaries, either at the beginning followed by space, at the end preceded by space or in between space characters.

But without word boundary operators, it gets more complicated when the words do not have to be adjacent:
grep -e '^$[a-z]\{1,\}$ $[^ ]* $*\1$' -e '^$[a-z]\{1,\}$ $[^ ]* $*\1 ' -e ' $[a-z]\{1,\}$ $[^ ]* $*\1 ' -e ' $[a-z]\{1,\}$ $[^ ]* $*\1$'
Another thing to note that this is just the case where words are on the boundaries with a space. But there can be comma's, semicolons punctuations etcetera.

--
If you have GNU or BSD grep (as opposed to standard grep) then you can use word boundaries as an extension to regex, so it can be simplified into something like this:
grep '\<$[a-z]\{1,\}$\>.*\<\1\>'
They also support back reference with extended regular expressions so, you can can do this:
grep -E '\<([a-z]+)\>.*\<\1\>'
Note in general instead of [a-z] , it is preferable to use [[:lower:]] for lowercase or [[:alpha:]] which matches both upper and lower case in all compliant code sets..

Thank you for the corrections Scrutinizer.

The grep command:

grep '\([a-z]+\)\1'

does include a valid BRE, but what it is looking for is a single character in the codeset of the current locale that is greater than or equal to a and less than or equal to z followed by a literal plus sign ( + ) followed by another copy of those same two characters.

In addition to using [a-z]\{1,\} to get one or more "a" through "z" characters, one can also use [a-z][a-z]* to match the same set of characters.

I had used a single space between words intentionally, because I thought that was what was wanted (i.e., adjacent duplicate words.) The BREs you used look for duplicated words on a single line whether or not they are adjacent.