How to remove repetitive lines in a file with sed?

Shell Programming and Scripting
1

Hello,
My goal is the make all x times repeated lines into a single line.
I need to attain the expected output with sed -i , I need to overwrite the MyFile
MyFile:

Hello World
Welcome
Hello World
Welcome Back
This is my test

Expected output:

Hello World
Welcome
Welcome Back
This is my test

I found below code in another forum while searching but it did not solve the issue, even with sed -n :

sed -n '
:start
/^\[/{
    h
  :loop
    n
    /^\[/b start
    /^$/b loop
    x;p;g
}
p' MyFile

It would be nice if you would help me out on getting rid of this issue.
Many thanks
Boris

2 
perl -i -ne 'print unless ${$_}++' MyFile
1 Like
3

Hello Aia,
Much appreciated, thank you..

Kind regards
Boris

4

Please make the request more specific: do you want to delete the

  • third line
  • first duplicate line
  • any duplicate lines
  • second "Hello World" line
    ?
5

Dear Rudic,
Thank you for your attention.
Aia's post helped me with this.
My file was consisting of 29000 similar lines and now it has only 55

perl -i -ne 'print unless ${$_}++' Filename

Many thanks
Boris

PS: I have just updated the main thread upon your notification

6

You wanted sed ? Try this; it works on your sample file, but I'm unsure how it would deal with above mentioned large file:

sed -rn '1h; 1!H; s/$/./; G; s/^(([^.]*)\.\n(.*\2.*))\2$/\3/; s/^[^.]*\.\n//; s/\n\n/\n/; h;  $p' file
Hello World
Welcome
Welcome Back
This is my test
7

Dear Rudic,
My main aim was to overwrite with sed -i , for that reason I kindly asked if that was possible to execute with sed command but Perl is also okay.
Regarding your sed command, seems like my file is not compatible with your scenario. The terminal is waiting for printing the output but no action.

Kind regards
Boris

------ Post updated at 02:47 PM ------

Dear Rudic,
Even though I said "sorted out" but I see that sometimes newly created files are not accepting sed or perl command for this issue and it asks me to convert newly created files to unix with dos2unix command. Pc is running under ubuntu 18.04 bionic
How may I get rid of this? Maybe this could be a new thread topic.

Thanks in advance
Boris