Yeah that's really not ok. You have every right to feel annoyed, worried, upset.

Following on from what Katie suggest you could try a sign on the door politely letting people know that visitors are not welcome atm (you don't have to say why, just that this isn't a good time to drop in) and ask people to call you instead? I've got a mate who had twins who did this when she needed time out and to know she wasn't going to have to entertain neighbours.

after what you've been through that was the last thing you needed. I'm sure your DD will be ok. Washing her face/hands etc was a good idea.