I think genetics play a pretty big role. My babies have been different weights, but all relatively small, regardless of how much weight I gained, or how far along I was when they were born (all between 38 and 40+1, although my 38 weeker is my smallest one, so that does have some impact for sure.)
A friend of mine had one boy who was huge at birth - big head and shoulders, shoulder dystocia, etc. Her girls were smaller, and she thought it was a boy/girl thing. But then, her second son was much more petite. One boy (and the girls) took after her side, the other took after his father. It can be really hard to predict and/or control!

As far as your tummy muscles go, statistically speaking, anyone who has had 2 babies will have some degree of muscle separation, and that can make your back and belly sore. If you're in pain, or worried about that starting, get a good belly support. I have a Blanqi pregnancy support tank that I've worn daily from about 12 or 14 weeks, and it has made a BIG difference, but I'm sure there are lots of good options you could look into. Exercise to strengthen your other muscles, and wear good shoes to help your back and hips out.
Hope that helps!