Understanding regex behaviour when using quantifiers

# echo "Teest string" | sed 's/e*/=>replaced=</'
=>replaced<=Teest string

So, in the above code , sed replaces at the start. does that mean sed using the pattern e* settles to zero occurence ? Why sed was not able to replace Teest string.

# echo "Teest string" | sed 's/e*//g'
Tst string

How does it work when global flag turned on ?

Some flavors of regex have + for one or more, but you can just say 'ee*'. Also, there is '\{1,99\}' for 1 to 99 in the sed flavor. There must be about a dozen regex flavors, especially after the PERL guys dominated a POSIX version, so the word edge '\>' became '\b': Regex Tutorial - \b Word Boundaries

There are even schemes to make the * lazy as opposed to the normal greedy behavior. Consider the ksh/bash ${pathname##/} is greedy, leaves just the entry name, but the ${pathname#/} just removes the first slash and anything before it. This is not a standard regex, but I recall MULTICS qedx having a way to do the agressive/lazy switch back when. I wonder if regex are older than UNIX?

The g says how many times to apply the substitution: infinite. You can also say 3 to skip to the third match before substituting. It has to do with the writing, not the matching. With no flag, same as 1. Regex Tutorial - Possessive Quantifiers

1 Like

Despite what DGPickett said, perl had no affect on the description of regular expression in the POSIX standards.

There are several variations on RE processing, but there are three main types (basic regular expressions [BREs], extended regular expressions [EREs], and pathname pattern matching) in the standards (POSIX, the Single UNIX Specification [SUS], the System V Interface Definition [SVID], and the Linux Standard Base [LSB]). According to POSIX, SUS and SVID, sed and a bunch of other utilities use BREs, awk and a bunch of other utilities use EREs, and the shell and a bunch of other utilities use pathname pattern matching when expanding pathnames. POSIX has about five full pages describing BREs, three and a half pages that describe the differences between BREs and EREs and another four and a half pages that give the formal grammar for the interpretation of BREs and EREs, and about two and half pages that describe the differences between pattern matching and REs.

Some utilities (like grep) have options to choose between BREs, EREs, and fixed strings. Although not specified by the standards, some implementations have options for sed to choose between BREs and EREs, but using EREs with sed is not portable.

Meanwhile, back to your question. In the BREs used in sed, the expression e* matches zero or more occurrences of an e . The beginning of the string Teest string (before the "T") matches zero occurrences of "e", so the pipeline:

echo "Teest string" | sed 's/e*/=>replaced=</'

produces:

=>replaced=<Teest string

and the command:

sed 's/e*/=>replaced=</g'

would replace every occurrence of zero or more "e"s with your replacement string. I.e.,

=>replaced=<T=>replaced=<s=>replaced=<t=>replaced=< =>replaced=<s=>replaced=<t=>replaced=<r=>replaced=<i=>replaced=<n=>replaced=<g=>replaced=<

Two portable sed pipelines to do what you were trying to do are:

echo "Teest string" | sed 's/e\{1,\}/=>replaced=</'

or (as DGPickett suggested):

echo "Teest string" | sed 's/ee*/=>replaced=</'

which replaces the first occurrence of one or more "e"s with the specified replacement string. In this case:

T=>replaced=<st string
2 Likes

There even more interesting aspects to regular expressions.
Consider reading the 'owl' book: Mastering Regular Expressions 3rd Ed by J. Friedl

Thanks, I got one already. Guess its good time to start reading that :slight_smile:

Thanks to all who for a brief explanation on this :slight_smile:

FWIW: The standards issues with regex are something that that appears to be coming together well. Or better anyway.

Basically when you are using UNIX tools, IMO, regex use has this sort of feel to it:

If today == Tuesday 
  then 
     we must be in Belgium
end if

This is the way UNIX was overall back in the 90's - XOPEN, SUS, SVID, SYSV, BSD, Torvalds etc.

Henry Spencer ( zoologist) wrote the first open source version of UNIX regex, which then allowed the creation of cascade of modern regex "flavors". Larry Wall appears to have used Spencer's regex as a model for perl regex, for example.

So, if you understand the difference between extended regular expressions (ERE) and basic (BRE) you are well on the way.... to Belgium.

This is shorter: Regular expression - Wikipedia, the free encyclopedia

QED went to MULTICS, probably before UNIX, and became qedx, which is very close to ed and ex. Later, Waterloo ported it to FRED.

grep -E is essentially egrep is ERE.