i have never used sshfs but i know ssh first attempt to authenticate a connection using public/private key authentication and then using password when the first one fail.
Does it still ask you for a password when you just connect to the server? with something like this from the client "ssh remote_user@server_address"
ssh asking for a password usually means your private/public key setup is not working correctly somewhere. Check again and make sure the user on client computer who wants to make a connection does so with a correct private key and the corresponding public key is on the correct account on the server's authorized key file