I still did 40 min of pretty high paced exercise 5 days a week, walked 12 blocks pushing the stroller to my kids swimming lessons, ran my household, etc. You're not overdoing it, necessarily.
But you might need a little help! Get some compression stockings, maybe, and a belly/back support. It will make a BIG difference.