At this age if the invitation is to the 4yr old only I would expect Mum or Dad and the party guest only.
If they needed to bring other kids due to not being able to have them looked after, then I would have thought they should ring you and check that it was ok.
We have just been to a couple of 4yr old parties, and it was just DS2 with me or his Dad while the other stayed at home with the kids.