It sounds like you and the other mothers just have a different focus on what you want to get out of the group. It doesn't make what you're doing "wrong". If you want to run an activity can you organise an outside game (like throwing balls) or something where you feel you can watch your son while doing the game?

I wouldn't worry too much about the group if it's not working out. Sometimes you just don't gel with people and that's okay. Go to the other group and do other activities like go to busy playgrounds where other children are and visit the library storytime etc.