Hun we had the same issues with DH fam. We told them early on the names we were considering and got some really rude comments. We now call bub by her name at home & DH accidentally let it slip in front of his mum and she is still trying to change our minds, makes me so cross.
If you love the name & don't use it because of fam comments you may regret it. If you love it go for it. I think Rafferty is an awesome name nn Raffy. Like I said I know a little boy Raffy & he's so cute!