You just get bigger sooner but not overall. That was my experience anyway. I was definitely bigger earlier with DS2, but I didn't grow as quickly and by about 6 months I was pretty much the same as my first pregnancy. In fact, in the last trimester people were telling me I wasn't very big.
Don't stress about belly size... take photos and you will laugh about having thought you were big when you look back.