Hi I have a file that are a list of people & their credentials i recieve frequently The issue is that whne I catnet this list that duplicat entries exists & are NOT CONSECUTIVE (i.e. uniq -1 may not weork here )
I'm trying to write a scrip that will remove duplicate entries
the script can typically made up of the following :
--------------
Ms AA
Unique to A
More of A
Mr BB
Mr CC
Ms AA
Unique to A
More of A
Mr DD
Mr EE
Mr BB
------------
Some of my technqiues of just are't working quite right especially with ignoring white spaces (maybe sed here)
(e.g. awk -F, '! mail[$3]++' inputfile )
any tips ?
ts
ms s
No worries
I've seen this forma ased manual somewhere...
# delete duplicate, consecutive lines from a file (emulates "uniq").
# First line in a set of duplicate lines is kept, rest are deleted.
sed '$!N; /^\(.*\)\n\1$/!P; D'
# delete duplicate, nonconsecutive lines from a file. Beware not to
# overflow the buffer size of the hold space, or else use GNU sed.
sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P'
ALSO
XXXXX '!($0 in a);{a[$0]=1}' logfile
#where XXXX= awkfor Linuz & NAWK for Solaris
anyone know how to integrate ignoring blank lines in the above scripts ?
bump ;
Any clever SED / *AWK'rs out there who knows how to ignore blank lines & can integrate into the above examples.....?
Ignore blank lines:
sed 's/^$//g'
# or
grep -v ^$
What is the expected output given your sample data?