I actually stopped going to playgroup for this reason, amongst others. For us, we had DDs preschool drop offs and pick ups clashing with nap times too so in the end, I decided to simplfy our days and it was the right choice for us.

Having said that, i am thinking about going back in the new year when DS will probably be down to one nap in the middle of the day which will make life a lot easier.

I say do whichever is better for you - if you think you'd stress too much about his sleep (and I truly understand this) then don't go. If you can tell yourself it's just once a week and you need to go out, then go.