# filtering out duplicate substrings, regex string from a string **URL:** \ **Category:** Shell Programming and Scripting\ **Created:** [June 15, 2010, 11:18am UTC](https://community.unix.com/t/filtering-out-duplicate-substrings-regex-string-from-a-string/267701 "2010-06-15T11:18:35Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![kchinnam](https://community.unix.com/letter_avatar/kchinnam/32/6_22734026db12803ebb3e059622ac3534.png) [@kchinnam](https://community.unix.com/u/kchinnam)\ **Post date:** [June 15, 2010, 11:18am UTC](https://community.unix.com/t/filtering-out-duplicate-substrings-regex-string-from-a-string/267701/1 "2010-06-15T11:18:35Z")
My input contains a single word lines. From each line > [@](#): > > a) I want to remove all text that starts with 'dp' including 'dp'. > Ex: prjgoodBlaBladpgoodBlaBla ---\> prjgoodBlaBla > b) Also I want to remove duplicate substrings. > Ex: prjtestBlaBlatestBlaBla ---\> prjtestBlaBla > Logic I have in mind but having hard time implementing: Take 4 thru 10 characters [testBla] , if its found in the string, remove all text starting from second occurance of it. data.txt ```plaintext prjtestBlaBlatestBlaBla prjthisBlaBlathisBlaBla prjthatBlaBladpthatBlaBla prjgoodBlaBladpgoodBlaBla prjgood1BlaBla123dpgood1BlaBla123 ``` Desired output --\> data\_out.txt ```plaintext prjtestBlaBla prjthisBlaBla prjthatBlaBla prjgoodBlaBla prjgood1BlaBla123 ``` I am able to get part a) of my requirement working using following,, ```plaintext > sed 's/dp\(.*\)\..*/\1/' data.txt prjtestBlaBlatestBlaBla prjthisBlaBlathisBlaBla prjthatBlaBladpthatBlaBla prjgoodBlaBladpgoodBlaBla prjgood1BlaBla123dpgood1BlaBla123 ``` but not part b). ---
**Author:** ![bartus11](https://community.unix.com/user_avatar/community.unix.com/bartus11/32/1418_2.png) [@bartus11](https://community.unix.com/u/bartus11)\ **Post date:** [June 15, 2010, 11:32am UTC](https://community.unix.com/t/filtering-out-duplicate-substrings-regex-string-from-a-string/267701/2 "2010-06-15T11:32:29Z")
```nohighlight perl -pe 's/dp.*// || s/(\w+)\1/\1/' data.txt ``` ---
**Author:** ![kchinnam](https://community.unix.com/letter_avatar/kchinnam/32/6_22734026db12803ebb3e059622ac3534.png) [@kchinnam](https://community.unix.com/u/kchinnam)\ **Post date:** [June 15, 2010, 12:07pm UTC](https://community.unix.com/t/filtering-out-duplicate-substrings-regex-string-from-a-string/267701/3 "2010-06-15T12:07:36Z")
bart, its working.. Thanks for the solution.. Can you explain what does || and (\w+) do ? Can we get it working using sed !? can someone help ? ```nohighlight > /usr/xpg4/bin/sed -e 's/dp.*//' -e 's/(\w+)\1/\1/' data.txt sed: command garbled: s/(\w+)\1/\1/ ``` ---
**Author:** ![bartus11](https://community.unix.com/user_avatar/community.unix.com/bartus11/32/1418_2.png) [@bartus11](https://community.unix.com/u/bartus11)\ **Post date:** [June 15, 2010, 12:14pm UTC](https://community.unix.com/t/filtering-out-duplicate-substrings-regex-string-from-a-string/267701/4 "2010-06-15T12:14:42Z")
I don't know how to make it work in sed. "||" works as "exclusive or" in perl so it checks if first command was successful, and if it was then second one is not processed. (\w+)\1 matches first occurance of consecutive duplicate strings (it is extension of "(\w)\1", which would match consecutive duplicate characters, like "aa","bb" and so on). ---
**Author:** ![kchinnam](https://community.unix.com/letter_avatar/kchinnam/32/6_22734026db12803ebb3e059622ac3534.png) [@kchinnam](https://community.unix.com/u/kchinnam)\ **Post date:** [June 15, 2010, 1:56pm UTC](https://community.unix.com/t/filtering-out-duplicate-substrings-regex-string-from-a-string/267701/5 "2010-06-15T13:56:03Z")
Thanks bart. Does anyone know how to do this using sed ? does any other shell in Unix recognize (\w+) as consecutive duplicate strings? ---
**Author:** ![bartus11](https://community.unix.com/user_avatar/community.unix.com/bartus11/32/1418_2.png) [@bartus11](https://community.unix.com/u/bartus11)\ **Post date:** [June 15, 2010, 2:02pm UTC](https://community.unix.com/t/filtering-out-duplicate-substrings-regex-string-from-a-string/267701/6 "2010-06-15T14:02:11Z")
> [@kchinnam](#): > > Thanks bart. > > Does anyone know how to do this using sed ? does any other shell in Unix recognize (\w+) as consecutive duplicate strings? It is not just (w+), but (w+)\1. That "\1" is important, as it matches string matched before by (\w+). In other words "\1" matches the duplicated part. ---
**Author:** ![kchinnam](https://community.unix.com/letter_avatar/kchinnam/32/6_22734026db12803ebb3e059622ac3534.png) [@kchinnam](https://community.unix.com/u/kchinnam)\ **Post date:** [June 15, 2010, 3:26pm UTC](https://community.unix.com/t/filtering-out-duplicate-substrings-regex-string-from-a-string/267701/7 "2010-06-15T15:26:24Z")
Does anyone know how to achieve this using sed ? ```nohighlight perl -pe 's/dp.*// || s/(\w+)\1/\1/' data.txt ``` ---
**Author:** ![vgersh99](https://community.unix.com/user_avatar/community.unix.com/vgersh99/32/14851_2.png) [@vgersh99](https://community.unix.com/u/vgersh99)\ **Post date:** [June 15, 2010, 3:53pm UTC](https://community.unix.com/t/filtering-out-duplicate-substrings-regex-string-from-a-string/267701/8 "2010-06-15T15:53:22Z")
> [@kchinnam](#): > > Does anyone know how to achieve this using sed ? > > perl -pe 's/dp.\*// || s/(\w+)\1/\1/' data.txt ```plaintext sed 's/dp.*//;s/\([^]*\)\1/\1/g' data.txt ``` ---
**Author:** ![kchinnam](https://community.unix.com/letter_avatar/kchinnam/32/6_22734026db12803ebb3e059622ac3534.png) [@kchinnam](https://community.unix.com/u/kchinnam)\ **Post date:** [June 15, 2010, 5:14pm UTC](https://community.unix.com/t/filtering-out-duplicate-substrings-regex-string-from-a-string/267701/9 "2010-06-15T17:14:49Z")
Thanks, its working.. ```nohighlight sed 's/dp.*//;s/\([^]*\)\1/\1/g' data.txt ``` s/dp.\*//; --\> remove string from each line that starts with 'dp' followed by any character. s/\([^]\*\)\1/\1/g --\> For any string/line that is not empty, print only first occurance of the string !?