How to remove duplicate lines?

Shell Programming and Scripting
1

Hi All,
I am storing the result in the variable result_text using the below code.

result_text=$(printf "$result_text\t\n$name")

The result_text is having the below text. Which is having duplicate lines.

file and time for the interval 03:30 - 03:45 
file and time for the interval 03:30 - 03:45 
file and time for the interval 03:45 - 03:59 
file and time for the interval 04:00 - 04:15 
file and time for the interval 04:00 - 04:15 
file and time for the interval 04:15 - 04:30 
file and time for the interval 04:15 - 04:30

I have tried the below code to remove duplicates but not working.

result_text=$(printf "$result_text" | sort | uniq)

Please help me to remove duplicate lines.
Thanks in advance.

2

Works for me, given you correct the variable assignment to include the last trailing space, and the text remains sorted as it is right now:

printf "$result_text" | uniq
file and time for the interval 03:30 - 03:45 
file and time for the interval 03:45 - 03:59 
file and time for the interval 04:00 - 04:15 
file and time for the interval 04:15 - 04:30
3

Hello nalu,

Your Input_file is having spaces at last of lines. Could you please try following and let me know if this helps you.

awk '{sub(/[[:space:]]+$/,"");} !A[$0]++'   Input_file
OR
echo "$result_text" | awk '{sub(/[[:space:]]+$/,"");} !A[$0]++'

Thanks,
R. Singh

4

Hi Ravinder,

Thanks for your prompt response.

This code is removing duplicate lines.

echo "$result_text" | awk '{sub(/[[:space:]]+$/,"");} !A[$0]++'

If I assign the result to a variable it's not working.

result_text1=$(echo "$result_text" | awk '{sub(/[[:space:]]+$/,"");} !A[$0]++')

echo $result_text1

Please help me.

Thanks in advance.

5

Hello nalu,

You could hit THANKS button for saying thanks to anyone here.
Try following and let me know if this helps you. I am getting proper output from this command.

result_text1=$(echo "$result_text" | awk '{sub(/[[:space:]]+$/,"");} !A[$0]++')
echo "$result_text1"

Thanks,
R. Singh

1 Like