I would leave the younger at home. This is something that will happen alot in the future. I've invited one child in a family to my DS' parties in the past because that's the child who he plays with.
I have no problem with it at all. I think it's great for the kids to have their own sets of friends ( plus we live in a SUPER isolated area, so the kids having time apart is not often and really really good for them).