Deleting Duplicates leaving the first entry

Shell Programming and Scripting
1

Hi,

I need to delete duplicate records in a file that is around 30MB. Below is what I need. Below are the entries of input file and the output file that I need. Each section of input file is separated by an empty line. Some of these sections have duplicate uid values. I want to retain only one UID per section. I do not want to make any changes to other entries though they are repeated. I tried some options but the values of d and e which are same in other sections are getting deleted. I have also attached a file with actual entries in it.

Thanks in advance
sam

Input file entries:

a: t1.com
uid: t1
b: t2
c: t3
uid: t1
d: t3
e: t4
 
a: u2.com
uid: u2
b: u2
c: t3
uid: u2
d: t3
e: t4
 
a: v2.com
b: v3
c: t3
uid: v3
d: t3
e: t4
 
Expected OutPut:
a: t1.com
uid: t1
b: t2
c: t3
d: t3
e: t4
 
a: u2.com
uid: u2
b: u2
c: t3
d: t3
e: t4
 
a: v2.com
uid: v3
b: v3
c: t3
d: t3
e: t4
2

nawk -f sam.awk myFile

sam.awk:

BEGIN {
  RS=FS=""
}
{
  uid=0
  for (i=1;i<=NF;i++) {
    if($i ~ "^uid:" && ++uid>1) continue
    printf("%s%c%c", $i, ORS, (i==NF)?ORS:"")
  }
}
3 
awk '/^dn:/{s=0} /^uid:/{if (s==0) {s++} else {next}}1' Example.txt
4

The one liner nawk script worked. Thanks a lot for help...

SAM

---------- Post updated at 07:58 PM ---------- Previous update was at 07:57 PM ----------

Thank you all for the replies :slight_smile:

---------- Post updated at 08:04 PM ---------- Previous update was at 07:58 PM ----------

I have one more question:

I am using the below script to add the uid value with cn value in a file. I have a file that is 28 mb. It is taking around 6 hours to precess the file. I am getting the expected result. Is there any other way we can expediate the same process with code changes.

while read LINE
do
  if [[ $(echo $LINE|awk '{print $1}') != "cn:" ]]
  then
    echo $LINE >> out.txt
  else
    echo $LINE >> out.txt
    echo $LINE | awk '{print "uid: "$2}' >> out.txt 
  fi
done < Test1.ldif

Thanks in advance
Sam

5

This should be faster:

while read LINE
do
  echo $LINE
  [[ $LINE == cn:* ]] && echo "uid: ${LINE##cn:* }"
done < Test1.ldif > out.txt

Or, if your sed supports \n :

sed 's/^cn: *\(.*\)/cn: \1\nuid: \1/' Test1.ldif > out.txt

Or, with awk:

awk '/^cn:/ { $0=$0"\nuid: "$2 } 1' Test1.ldif >out.txt