I think that until people become your friends, as opposed to acquaintances, it's always going to be difficult. You will either be working to make sure that you are being proactively polite so no-one can possibly be upset, or you will be getting offended when some little kid keeps smacking your bub in the face and the mum doesn't seem to care or intervene.

But when you become friends, then it's more relaxed because you have the floor to say 'hey hon, did you know your bub's whacking my kiddo' and they know you think they're a good parent so don't take it badly, or you can say to their child 'hey buddy, can you stop hitting xx' and they're not too fussed because they know you respect their child. And even if you do offend each other, friends are usually in the position to communicate that in a friendly way, so that next time you can do things differently to keep the friendship.

So I guess if you wanted, you could try to build on the friendship side of things, maybe suggest just the mums get out for dinner or something? Otherwise, I think just accept that things are as they are and be grateful that things are erring on the polite side as opposed to the bullying side.