if he doesnt go i dont think you'll be labelled, just RSVP saying thanks but u already have plans.
FWIW i think 3 is a bit young for a macca's party