I expect people to tell me if they have a cold, and if they have then to stay away till they're better. They'll get their chance to see the new baby. I was the victim of undisclosed sickness and had a baby who was sick and fluey for about 6 weeks from 2 weeks of age. Her attachment has never been the same since those first two weeks, because the havoc on her sinuses tightened up her jaw muscles.
I only allowed my partner and DS to hold DD, after my wishes when DS was born were disregarded. But S&BiL didn't think to keep their snotty DD from leaping over and planting a snotty kiss on my 2 week old baby. I hope to have no offended faces if I go again and ban close contact for 4 weeks!
I'd be wearing bubs in a close fitting sling, like a HAB, for four weeks, so that the people who come near are people you can be bothered taking her out for, and people you feel confident don't have lurgies.
You regret letting people hold more than you regret NOT letting them hold. Believe me.