It's ok to simply not like it, but seriously, there is really nothing that your kids will catch from a simple touch of someone's hand to their arm or even cheek.
Actually they are a few thing you can catch from a simple touch. Especially an immune compromised baby.
When I didn't want people touching my prem baby I would cover the pram with a blanket or sun-cover.