Remove duplicate email

UNIX for Beginners Q & A
1 
cat path/to/dir/file.html | grep -i 'x*.com' > path/to/dir/file.txt

Before
xyz.com 
xyz.com

After 
cat path/to/dir/file.html | grep -i 'x*.com' |  sed '$!s/$/,/' | tr -d '\n'> path/to/dir/file.txt

Result --> Preferred
output:
xyz.com, xyz.com

The preferred is the exact output I want, but I want to remove duplicate. I tried sort | uniq but still wont work. Any help appreciated.

2

how about (for starters):

awk -v str='x*.com' '$1~str && !a[$1]++' myFile
1 Like
3

Sorry still show duplicate

4

myFile:

xyz.com
xyz.com
1xyz.com
123.com

$ awk -v str='x*.com' '$1~str && !a[$1]++' myFile
xyz.com
1xyz.com
123.com
5

I need them to be on the same line with comma seperated as it appears in my preferred output.

6

Your grep seems not precise, perhaps you mean grep -i '^x.*\.com$' (starts with an x then any amount of characters then .com at the end)
A sed 's/$/,/' does not delete $ because it is an anchor - not a character. But after an N command (that appends the following line to the input buffer) one can remove the embedded \n character.
The following works on all Unix-like OS:

grep -i '^x.*\.com$' file.html | sed -e ':L' -e '$!N;s/\n/, /;tL'
7

This removes the emails that are the same. I need to only remove the duplicates

8

@vgersh99
Got your script to work but need to add sed '$!s/$/,/' | paste -sd ""

 awk -v str='x*.com' '$1~str && !a[$1]++' myFile | sed '$!s/$/,/' | paste -sd ""

to get my required output. Not sureif thats the best way, but I got it to work. Advice?

Thank you

9

A clunky way:-

grep -Ei "x.*\.com" /path/to/dir/file.html | sort -u > /path/to/dir/file.txt

The expression looks for an x followed by any number of characters followed by .com however this is not anchored to the beginning or end of a line. What is your input data like?

This input would still give some confusing results:-

x123@hello.com
x123@hello.com.foo
hello1@xyz.com
hello2@xyz.com
hello1@not-xyz.com-either

..... and lots of other variations. It leaves me a few questions::-

  • What precise conditions do you want for the search in the first place?
  • What output do you want? The full email address or just the domain.

We are adjust the search to get just records you are after, but the search needs to be precise, e.g. does the line start with x or have x immediately after @ ; does .com have to end the line etc. All sorts of rules can be written if you can be sure what you want. If you could post a representative sample of your input and desired output (in CODE tags) then that will give us more to work with.

Kind regards,
Robin

10

You can use

awk '!a[$1]++' myFile | paste -sd "," -

If you want to replace each newline with ", " then

awk '!a[$1]++' myFile | sed -e ':L' -e '$!N;s/\n/, /;tL'