Ok...maybe i'm being a little out of line and I hope I don't offend anyone.
But... Like i said before Maybe she thought you'd met before. Maybe it was just a slip of the mind....

Yes you have EVERY single right to be worried about your childs safety.
However if you are concerned and are worried about it...i would simply ask who it was. How do you know it wasn't just that, an uncle, did you check the visitors book to see if yes they signed in or out??? Was the carer with the guy...he wasn't left alone with the kids???
I don't feel that you can judge what a carer is like based off this one experience, especially if you haven't clarified any of the details yourself.

My FDC mum has her parents there on occasion and also her grown up kids. While they get more one on one attention in FDC i believe it is also good to expose them to other people/visitors.

I just think that until you have ALL the details (eg you ask and find out who it is) you shouldn't make any huge decisions that could later have an impact upon your son.

PS. Like i said in the begining...this is my opinion and if offend anyone i'm sorry.