I think if you only want your DH there then go for it, you have to do what you feel comfortable with. What works for some might not be the best for you, especially if you butt heads with your family in day to day life.
I had my Mum and DP there, although my Mum was vomiting in the shower almost the whole time so didnt really end up being much support.