I think you need to generate a key pair on the other server and use that key pair to go in the other direction. You can use the same key pair on each server but you need to copy it over to the other server. I do it this way, because I have a automount share, so I have the same home directory on all the same servers.
It doesn't work both ways. To connect in both directions, you need to generate keys and add to authorized_keys in both directions. Two different keys, separately generated for each way.