It's not you it's them. And changing who you are in order to fit in will never end well.
Change groups, I would go to the one where you feel more comfortable, even if it is harder to get to.