I'm sure others have dealt with this in the past as have I, but it remains problematic so I'm after other opinions...
When I connect with SSH/SFTP to a remote server, the remote host fingerprint is stored in the known_hosts file. If that remote host is the DNS alias (CNAME) for the production server in a production/contingency pair the entry in known_hosts is the production server's fingerprint logged against the DNS alias.
Now, when that remote system switches to its contingency server, the CNAME will follow it and the next SFTP connection sees a different fingerprint and thinks it's a 'man in the middle' attack.
I've resolved this from the server side previously by copying the hostid from production to contingency so both servers report the same fingerprint and systems connecting to me switched seamlessly. Now as the client, I can't force the remote systems to share their hostid's so I'm after a way of managing a remote switch without manual intervention to resave a fingerprint.
Options:
- I think I can tell SFTP to ignore the fingerprint, but that really does leave the attack door open.
- I think I was told several years ago that a 'composite' fingerprint could be created combining all of the remote fingerprints into a single entry, but I've never done it. Does anyone know how?
- Anything else?
So the failover pair need to have the same SSH server keys (probably in /etc/ssh) and it all works fine. You can just copy the files from one to the other, however, make sure you have several active sessions so you can undo it if testing fails (else you may not be able to log in to fix it)
Make sure you test it immediately.
I hope that this helps,
Robin
That indeed works nicely if the server people want to play ball but my experience is that asking them to copy keys from one server to another gets short shrift.
I have now discovered that the OpenSSH known_hosts file can very easily be edited...
process is to connect to each server explicitly in the failover pair so their host keys get added to known_hosts. Then edit the file to add the DNS name that you want to use into field #1 on each of the host records. Or alternatively, copy the host records to two new records and change field #1 completely to the DNS name i.e. two entries with different names but with the same key.
If you're using Tectia, it's more complicated - you need to concatenate the individual key_* files for the individual servers into a new file called key_22_[DNS].pub
1 Like
I'm not sure what is the issue with copying host keys.
Those many systems are actually represented as one (VIPs, CNAME ..) so all should be fine security wise.
If you do not want to copy those keys, you can keep them on shared storage e.g NFS, again same for all hosts, even prepared earlier.
Problem (not a big one tho) would arise if those systems are used for other services which are accessed directly not behind load balancer or similar logic.
But this would require to rethink to separate the service in containers for instance, or distinct VMs to further isolate / secure the stuff.
When securing stuff, start with network, e.g host or standalone firewalls to limit access from outside VLANs from specific stuff in your network, then everything else (keys, 2FA, user policies etc.), followed by an audit system in place with log shipment, depending how deep you want to go and information being held by those services.
Hope that helps.
Regards
Peasant.