OK, here is another 'take' on it.
You are not much of a friend to him if you cannot accommodate his fears when he wishes to visit you.
If he had a per pet lion/tiger/whatever in his home, and you said you feared it, would you expect him to accommodate you to allow you to visit him?
....... just another way of looking at it ...... 
edit for spelling
If I had a friend that owned a Tiger. And I trusted that friend. And he assured me that the tiger was tame and safe. And he could illustrate it to be so by playing safely with kids and himself and other strangers on the property, all of which I can do with my dogs,, I would trust him.
In fact, I would jump at the chance to play with a tame tiger.
Your point however is well taken.
If someone claimed to have a tame
crocodile that I could pet and play with, I might have second thoughts.