Remove duplicate consecutive lines with specific string

Shell Programming and Scripting
1

Hello,

I'm trying to remove the duplicate consecutive lines with specific string "WARNING".

File.txt

abc;
WARNING 2345
WARNING 2345
WARNING 2345
WARNING 2345
WARNING 2345
bcd;
abc;
123
123
123
WARNING 1234
WARNING 2345
WARNING 2345
efgh;
1234
1234
12
134
WARNING 2345
WARNING 1234
WARNING 1234

Output should be something like this -
out.txt

abc;
WARNING 2345
bcd;
abc;
123
123
123
WARNING 1234
WARNING 2345
efgh;
1234
1234
12
134
WARNING 2345
WARNING 1234

I have tried sed but didn't accomplish what I'm trying to achieve -

sed -r 'N; /(WARNING)[^\n]*\n\1/ s/\n.*//; P; D' File.txt
abc;
WARNING 2345
WARNING 2345
WARNING 2345
bcd;
abc;
123
123
123
WARNING 1234
WARNING 2345
efgh;
1234
1234
12
134
WARNING 2345
WARNING 1234

Please provide your suggestions to do that using sed or any other method.

Thank you.

2

Hello Mannu2525,

Could you please try following and let me know if this helps you.

awk 'NR==1{print;last=$0;next} ($0 != last){print}  {last=$0}'   Input_file

EDIT: Adding one more solution on same too now.

awk '{printf("%s",NR==1?$0 RS:(($0 != last)?$0 RS:""));last=$0}'

EDIT2: If you want to remove sequential only duplicates of lines which have string WARNING then following may help you in same too(with GNU awk ).

awk '/WARNING/ && !a[$0]++{print;next} !/WARNING/{delete a;print}'   Input_file

Thanks,
R. Singh

3

Able to do it using

sed '/WARNING/N;/^\(.*\)\n\1$/!P; D' File.txt
1 Like
4

Thanks Mannu, we appreciate when members let us know they have solved or found a solution and even more when showing us how, for the good of all the community

5

Another awk solution

awk '{if($1=="WARNING" && $0==dup){next}} {print;dup=$0}' file
6 
awk '$0!=dup; {dup=_} /WARNING/{dup=$0}' file
1 Like
7

This is really tricky.
With a Unix sed you also have the strange behavior that N exits without a default print, if it's on the last line (i.e. fails); the usual work-around is $!N .
Also many Unix sed need { and } (and lables) on separate lines or separate -e arguments.

sed -e '/^WARNING/{' -e '$!N;/^\(.*\)\n\1$/!P;D' -e '}' File.txt