I'd have said something to them, even if it was a simple "it's okay, I don't mind them coming near me" - and if it continued, I would eventually say some incredibly not-nice things to them