I think you just need to do what's best for YOU, not them. I went home the day after Emily was born, and as it was my birthday that day, I had all my family and my DH's family around for dinner. They couldn't believe how organised I was, and how I was really keen on having them all there! But that's what I wanted.
With Nina though, I didn't have a very good recovery at first, so I had to ask friends not to come until I was home and settled.

You might feel differently once bub is here, or you may not. Just evaluate things once bub is here and see how you feel.