Removing short strings in a particular column

UNIX for Beginners Q & A
1

I have a TAB SEPARATED file with two columns like this:

original	flattened
"H. H. Smith	H H Smith
(A. Adams, 1861)	A Adams 1861
(A. E. Verrill, 1885)	A E Verrill 1885
(A. Solari et F. Solari, 1903)	A Solari et F Solari 1903
(A.J. Wagner, 1909)	AJ Wagner 1909
(Abeille de Perrin, 1904)	Abeille de Perrin 1904
(Affa'a, 1986) Affa'a in Aescht, 2001	Affaa 1986 Affaa in Aescht 2001
(Agass. MS. )	Agass MS 
(AH Clark, 1912) non Carpenter, 1888	AH Clark 1912 non Carpenter 1888
(Ahmad, 1985) Madhavi, 2011 nec Ahmad, 1985	Ahmad 1985 Madhavi 2011 nec Ahmad 1985

Now I want to replace all the short strings that are 3 characters or less from the second column with a space:

original	flattened
"H. H. Smith	Smith
(A. Adams, 1861)	Adams 1861
(A. E. Verrill, 1885)	Verrill 1885
(A. Solari et F. Solari, 1903)	Solari Solari 1903
(A.J. Wagner, 1909)	Wagner 1909
(Abeille de Perrin, 1904)	Abeille Perrin 1904
(Affa'a, 1986) Affa'a in Aescht, 2001	Affaa 1986 Affaa Aescht 2001
(Agass. MS. )	Agass 
(AH Clark, 1912) non Carpenter, 1888	Clark 1912 Carpenter 1888
(Ahmad, 1985) Madhavi, 2011 nec Ahmad, 1985	Ahmad 1985 Madhavi 2011 Ahmad 1985

I have tried the following:

awk -F"\t" '{OFS="\t";gsub(/\b.{1,3}\b/," ",$2);print $0;}' infile > outfile
awk -F"\t" '{OFS="\t";gsub(/\y.{1,3}\y/," ",$2);print $0;}' infile > outfile
awk -F"\t" '{OFS="\t";gsub(/\<.{1,3}\>/," ",$2);print $0;}' infile > outfile
awk -F"\t" '{OFS="\t";gsub(/[\s^].{1,3}[\s$]/," ",$2);print $0;}' infile > outfile

None of them seem to do anything. What am I missing?

2

do you mean 'words' that are 3 characters or less or any three characters ,
also removing , ( ) ' "

looking for clarity ...
what about ?names such as

Davis Love 3rd
Jerry Lee Lewis
Ryan O'Neill
Noah's Ark
Edwin Van Der Sar
Eric the Red
Eddie Van Halen
Bruce Lee
Courtney Cox
Mac'Ill'Anndrais
etc etc
.....

3

Thanks for your response. I am looking to remove any sequence of 3 characters from the second column. , ( ) ' " have already been removed, they are only in the first column. My desired result based on your example would be:

Davis Love
Jerry Lewis
Ryan O'Neill
Noah's
Edwin
Eric
Eddie Halen
Bruce
Courtney
Mac'Ill'Anndrais

Thanks!

4

as a debugging effort (for ease of visual identification), I tried to substitute each string where length<=3 substitute the string with itself surrounded by <>:

awk -F'\t' '{gsub(/(^|\s+)\S{1,3}($|\s+)/, "<&>", $2)}1' OFS='\t'  infile

I get everything correct, BUT the 2 sequential candidates:

original        flattened
"H. H. Smith    <H >H Smith
(A. Adams, 1861)        <A >Adams 1861
(A. E. Verrill, 1885)   <A >E Verrill 1885
(A. Solari et F. Solari, 1903)  <A >Solari< et >F Solari 1903
(A.J. Wagner, 1909)     <AJ >Wagner 1909
(Abeille de Perrin, 1904)       Abeille< de >Perrin 1904
(Affa'a, 1986) Affa'a in Aescht, 2001   Affaa 1986 Affaa< in >Aescht 2001
(Agass. MS. )   Agass< MS >
(AH Clark, 1912) non Carpenter, 1888    <AH >Clark 1912< non >Carpenter 1888
(Ahmad, 1985) Madhavi, 2011 nec Ahmad, 1985     Ahmad 1985 Madhavi 2011< nec >Ahmad 1985

I can't quite get the the correct regex for H H Smith or A E Verrill 1885 for example.
Need to polish the regex a bit more.

5

This is close to your requirement. (For some reason, GNU awk does not take \b as "word boundary".)
To get rid of some spaces, make the substitution string empty. And you can consume a trailing space with /\<.{1,3}\> */
A bit optimized it becomes

awk 'BEGIN{FS=OFS="\t"} {gsub(/\<.{1,3} +/,"",$2); print}' infile >outfile

A short word at the very end is not removed, this would require

awk 'BEGIN{FS=OFS="\t"} {gsub(/\<.{1,3}( +|$)/,"",$2); print}' infile >outfile

The word boundaries are markers only, so a match does not consume too much for a subsequent match (that starts to the right of the previous match).
But a "word boundary" applies to other no-word characters like dash or colon, so it's important that you removed them.

Perl can use a "look-ahead" space, that is a more precise word boundary marker. The following seems to work (using a look-behind):

perl -F"\t" -alne '$F[1] =~ s/(^|(?<= ))[^ ]{1,3} +//g; print join("\t", @F)' infile > outfile

Also remove a short word at the end:

perl -F"\t" -alne '$F[1] =~ s/(^|(?<= ))[^ ]{1,3}( +|$)//g; print join("\t", @F)' infile > outfile
6

Thank you both for your responses. Unfortunately, none of your AWK solutions work for me. Nothing in the second column changes in the outfile. It must be something weird in my system. I'm using a iMac running Sonoma 14.6.1. On the bright side, your perl solution works great, so I am going to use that one for now and maybe learn a little perl in the process.

7

@kaktus
just curious, what do you get if you run gawk --version?

8

Well, removing "any sequence of 3 characters" or less means removing the entire 2nd column. You really need to be more specific here.

9

Sorry, I didn't your message earlier. I just tried gawk, and it totally worked. Very strange. I use awk all the time and usually don't encounter any problems, although I haven't used gsub much. Thanks for the tip.

10

I meant "any sequence of 3 characters or less bordered by whitespace." It looks like other respondents understood that, and I now have two viable solutions (perl & gawks). Thanks all!