Hugs to you hun, I don't know how to say it nicely - but hey, should you have to?
I have 4 kids, and I would rather not take them to someone's house if there are sick kids there...and I always tell them if mine are sick too.
It's just manners. And to imply that you are wrong somehow for not wanting to expose your LO ... that is not manners.

I can't think of any snappy comebacks right now, I'm sure someone else will but I just wanted to let you know IMO you're perfectly normal