I second what CrazyCat said, just don't answer the door if it's not convenient for you. Any way you can see who it is before you answer it? If so, have a sneak peek, and if it's them, or any other unwanted guests, stay quiet until they leave. Next time you see them and they say they popped in and you weren't there, tell them they should have called first. Also you might want to say that you're telling all your friends and family to call before coming over in preparation for the baby.

Hth, I know how annoying it can be