I agree with Trav. Either invite them all or invite a handful.

ETA - and yes to *Ang*'s comment re meanness. Mean can be very subjective at that age. Generally when DS complains about another child being mean he means that they wouldn't play Harry Potter when he asked them to or something equally inoffensive. There are feral 4 year olds but I think it is better to delve into the issue than just accept 'mean' at face value.