DS is two in May and I've already started thinking about this. His 1st b'day was a 'we survived 12 months' kinda thing so we invited everyone who's been with us along the way, mainly adults. He also had a joint 1st birthday with my mother's group. this year it'll be his 'friends' only - those he has play dates with. Or, we'll just do a family party and leave the 'friends' party til he's 3 or older, when he actually has friends and knows what it all means.
If you didn't have a 1st b'day, you might want this to be an 'all in' kinda thing (i.e. it's a party for you too), but maybe just make it small nibbles and just a couple of hours. BYO drinks too. We found a lot of people didn't come to ours so our list of about 45 ended up being just over 25.
If you want it to be just for him, then it's ok to leave out YOUR friends that don't have kids, or even your friends that have older kids, and just invite the kids his age who he actually plays with. Address the invite to the child and if anyone is offended, explain what you're doing - e.g. You can come and help supervise, but there won't be any adult food/drink and it's all two year olds and tanties, lol!!!
Bookmarks