This happened a lot when DS was tiny. At 6 days old, an old lady stopped us in the shopping centre and asked how old he was. I said 6 days, and she reached out and touched his face. I instinctively jerked away from her and stepped backwards. I simply said "Ok, have a lovely day, bye" and off we went.

When out with him in the pram, I would drape a light cloth over the pram to deter strangers hands from entering. Worked a charm!