I was the only bridesmaid with two groomsmen when my cousin got married and it was fine. I walked back down the aisle with one on each side. During the bridal waltz, we each danced with our own parters and not other members of the bridal party. I preferred this so much anyway. The photographer was also really great and positioned us well so all the photos still looked really well balanced. It is not that difficult to do at all. I'm sure it will be fine!

ETA: Walking into the reception, we walked in together, I just had one GM on each side.