I haven't read all of the other replies but I thought I'd give you my honest opinion.

If it were me I'd take him out. IMO three year olds need the security of their mum more than they need socialisation. JMO though!

Is there a playgroup you could go to?
All the best with your decision