Delete duplicates in CA bundle

diva_thilak · June 28, 2014, 5:54pm

I do have a big CA bundle certificate file and each time if i get request to add new certificate to the existing bundle i need to make sure it is not present already. How i can validate the duplicates.

The alignment of the certificate within the bundle seems to be different.

Example:
Cert 1

MIIF7DCCBNSgAwIBAgIQbsx6pacDIAm4zrz06VLUkTANBgkqhkiG9w0BAQUFADCByjELMAkGA1UE
BhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQLExZWZXJpU2lnbiBUcnVzdCBO

Cert 2

MIIF7DCCBNSgAwIBAgIQbsx6pacDIAm4zrz06VLUkTANBgkqhkiG9w0BAQUFADCB
yjELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQL

These are duplicates and since they are aligned differently i could not do an extract and compare.

Your suggestions and inputs are most welcome.

Don_Cragun · June 28, 2014, 6:25pm

diva_thilak:

I do have a big CA bundle certificate file and each time if i get request to add new certificate to the existing bundle i need to make sure it is not present already. How i can validate the duplicates.

The alignment of the certificate within the bundle seems to be different.

Example:
Cert 1
MIIF7DCCBNSgAwIBAgIQbsx6pacDIAm4zrz06VLUkTANBgkqhkiG9w0BAQUFADCByjELMAkGA1UE
BhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQLExZWZXJpU2lnbiBUcnVzdCBO
Cert 2
MIIF7DCCBNSgAwIBAgIQbsx6pacDIAm4zrz06VLUkTANBgkqhkiG9w0BAQUFADCB
yjELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQL
These are duplicates and since they are aligned differently i could not do an extract and compare.

Your suggestions and inputs are most welcome.

If we ignore the line breaks, we can see that these two certs start with the same strings. But, the 1st cert ends with the characters:

ExZWZXJpU2lnbiBUcnVzdCBO

that do not appear in the 2nd cert. I don't see how you can say these certs are the same when one of them contains 24 more characters than the other one.

diva_thilak · June 28, 2014, 6:32pm

Thank you Don for your quick reply.

I have tried to post the first two lines to highlight the alignment issue. Yes you are right, removing the line breaks the certificates would be same.

Any suggestions on getting the duplicates removed from the file and direct them to a different file to show the number of duplicates.

Don_Cragun · June 28, 2014, 8:57pm

With what you have given us, I have absolutely no idea what you are trying to do.

What is a bundle?

Are both of these certs (and a bunch of others stored in a single file? If so, how are the certs delimited?

Do you have a bunch of files that each contain a cert?

Give us something concrete to work with! Show us sample input files to be processed. Explain the format of those input files. Show us the output that should be produced for those input files. Explain the format of the outputs to be produced.

diva_thilak · July 14, 2014, 11:40pm

In informatica server for authentication purposes the list of certificates are stored in a file. It is called as Certificate bundle.

They are stored in this format. My goal is to identify if there are any duplicates within this file (bundle). Since the alignment of the text within the begin and end tag are different, I was looking for an approach for identifying the dupes.

how to get the text between the begin and end tags extract
How to identify the dupes by bringing the text from step 1 in uniform format

CarloM · July 15, 2014, 7:39am

What's your OS? (or more specifically, do you have GNU awk?)

RudiC · July 15, 2014, 2:41pm

Try

sed -n '/CERTIFICATE/ {s/.*//g;x;s/\n//g;p;d}; H ' file | sort | uniq -c

and tell us the results.

Don_Cragun · July 15, 2014, 8:04pm

As long as the length of your certificates is less than the limit LINE_MAX bytes on your system (the limit on your system can be found using:

getconf LINE_MAX

but should be at least 2048), the following will reformat your certificates to 72 characters per line, tell you how many times each certificate appeared in your input file, and tell you where any duplicates appeared in your input file:

#!/bin/ksh
awk '
BEGIN {	nl = 72	# Normalized Length of certificate data.
}
/-----BEGIN CERTIFICATE-----/ {
	incert = 1	# We are processing a certificate.
	ac[++cc] = 1	# Increment # of certifcates seen and set appearance
			# count for this certificate
	next
}
/-----END CERTIFICATE-----/ {
	# End of certificate data found; look for a match...
	for(i = 1; i < cc; i++)
		if(l == l[cc] && cd == cd[cc]) {
			# We found a matching certificate.
			printf("# %s %d dups certicate #%d\n",
				"Certificate ending on line", NR, i)
			ac++		# Increment appearance count.
			cd[cc] = ""	# Delete this duplicate...
			l[cc--] = 0
			break
		}
	incert = 0	# We are between certificates.
	next
}
incert {# Gather certificate data.
	l[cc] += length($0)
	# NOTE: The following line will not work on some systems if certficate
	# data is more than LINE_MAX bytes long!
	cd[cc] = cd[cc] $0
}
END {	# Print results:
	for(i = 1; i <= cc; i++) {
		printf("# Certificate #%d, %d characters, appeared %d time%s\n",
			i, l, ac, ac[1] > 1 ? "s" : "")
		print "-----BEGIN CERTIFICATE-----"
		for(j = 1; j <= l; j = j + nl)
			print substr(cd, j, nl)
		print "-----END CERTIFICATE-----"
	}
}' "${1:-certificates.txt}"

If you want to try this on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk , /usr/xpg6/bin/awk , or nawk .

This was tested using the Korn shell, but will work with any shell that handles POSIX standard variable expansions. If the file certificates.txt (or another file of your choosing passed as the 1st operand to this script) contains:

the output will be something like:

# Certificate ending on line 53 dups certicate #1
# Certificate ending on line 69 dups certicate #2
# Certificate ending on line 99 dups certicate #2
# Certificate #1, 1036 characters, appeared 2 times
-----BEGIN CERTIFICATE-----
MIIDAzCCAmygAwIBAgIBATANBgkqhkiG9w0BAQQFADCBuzELMAkGA1UEBhMCQkUxETAPBgNV
BAcTCEJydXNzZWxzMRMwEQYDVQQKEwpCZWxTaWduIE5WMTgwNgYDVQQLEy9CZWxTaWduIE9i
amVjdCBQdWJsaXNoaW5nIENlcnRpZmljYXRlIEF1dGhvcml0eTElMCMGA1UEAxMcQmVsU2ln
biBPYmplY3QgUHVibGlzaGluZyBDQTEjMCEGCSqGSIb3DQEJARYUd2VibWFzdGVyQGJlbHNp
Z24uYmUwHhcNOTcwOTE5MjIwMzAwWhcNMDcwOTE5MjIwMzAwWjCBuzELMAkGA1UEBhMCQkUx
ETAPBgNVBAcTCEJydXNzZWxzMRMwEQYDVQQKEwpCZWxTaWduIE5WMTgwNgYDVQQLEy9CZWxT
aWduIE9iamVjdCBQdWJsaXNoaW5nIENlcnRpZmljYXRlIEF1dGhvcml0eTElMCMGA1UEAxMc
QmVsU2lnbiBPYmplY3QgUHVibGlzaGluZyBDQTEjMCEGCSqGSIb3DQEJARYUd2VibWFzdGVy
QGJlbHNpZ24uYmUwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAMQuH7a/7oJA3fm3LkHV
ngWxWtAmfGJVA5v8y2HeS+/+6Jn+h7mIz5DaDwk8dt8Xl7bLPyVF/bS8WAC+sFq2FIeP7mdk
rR2Ig7tnn2VhAFgIgFCfgMkx9iqQHC33SmwQ9iNDXTgJYIhXAs0WbBj8zfuSKnfQnpOjXYhk
0Mj4XVRRAgMBAAGjFTATMBEGCWCGSAGG+EIBAQQEAwIABzANBgkqhkiG9w0BAQQFAAOBgQBj
dhd8lvBTpV0BHFPOKcJ+daxMDaIIc7RqMf0CBhSZ3FQEpL/IloafMUMyJVf2hfYluze+oXkj
yVcGJXFrRU/49AJAFoIir1TqMij2De6ZuksIUQ9uhiMhTC0liIHELg7xEyw4ipUCJMM6lWPk
k45IuwhHcl+u5jpaR9Zxxp6aUg==
-----END CERTIFICATE-----
# Certificate #2, 1084 characters, appeared 3 times
-----BEGIN CERTIFICATE-----
MIIDKTCCApKgAwIBAgIENnAVljANBgkqhkiG9w0BAQUFADBGMQswCQYDVQQGEwJVUzEkMCIG
A1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMREwDwYDVQQLEwhEU1RDQSBFMTAe
Fw05ODEyMTAxODEwMjNaFw0xODEyMTAxODQwMjNaMEYxCzAJBgNVBAYTAlVTMSQwIgYDVQQK
ExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xETAPBgNVBAsTCERTVENBIEUxMIGdMA0G
CSqGSIb3DQEBAQUAA4GLADCBhwKBgQCgbIGpzzQeJN3+hijM3oMv+V7UQtLodGBmE5gGHKlR
EmlvMVW5SXIACH7TpWJENySZj9mDSI+ZbZUTu0M7LklOiDfBu1h//uG9+LthzfNHwJmm8fOR
6Hh8AMthyUQncWlVSn5JTe2io74CTADKAqjuAQIxZA9SLRN0dja1erQtcQIBA6OCASQwggEg
MBEGCWCGSAGG+EIBAQQEAwIABzBoBgNVHR8EYTBfMF2gW6BZpFcwVTELMAkGA1UEBhMCVVMx
JDAiBgNVBAoTG0RpZ2l0YWwgU2lnbmF0dXJlIFRydXN0IENvLjERMA8GA1UECxMIRFNUQ0Eg
RTExDTALBgNVBAMTBENSTDEwKwYDVR0QBCQwIoAPMTk5ODEyMTAxODEwMjNagQ8yMDE4MTIx
MDE4MTAyM1owCwYDVR0PBAQDAgEGMB8GA1UdIwQYMBaAFGp5fpFpRhgTCgJ3pVlbYJglDqL4
MB0GA1UdDgQWBBRqeX6RaUYYEwoCd6VZW2CYJQ6i+DAMBgNVHRMEBTADAQH/MBkGCSqGSIb2
fQdBAAQMMAobBFY0LjADAgSQMA0GCSqGSIb3DQEBBQUAA4GBACIS2Hod3IEGtgllsofIH160
L+nEHvI8wbsEkBFKg05+k7lNQseSJqBcNJo4cvj9axY+IO6CizEqkzaFI4iKPANo08kJD038
bKTaKHKTDomAsH3+gG9lbRgzl4vCa4nuYD3Im+9/KzJic5PLPON74nZ4RbyhkwS7hp86W0N6
w4pl
-----END CERTIFICATE-----