How is ssh from server A to server B, setup. Did you setup this server wide or user wide. In other words, did you setup a system key or user key for user 1?
Then I'm pretty sure the private id_rsa is different for the two users. Make sure either id_rsa is the same on server A or serverA:user1's public id_rsa.pub is appended to serverB:user2's authorized_key and vice versa.
I tried with the public key.. still it is not working...
that is I copied ServerA: user1 id_rsa.pub key and added in ServerB:User2 authorizedLkey file.
Also I tried
copying ServerA:User2 id_rsa.pub and added in ServerB:User1 auth key file.