Go through your wardrobe, sort it into 3 piles.

Smart - not really good for everyday, eg ball dresses.
Nice - dressy uppy and wearable, eg longer flowy skirts, stretch dresses that allow feeding, nice blouses, T-shirts, smart jeans, fashionable trousers.
Not outside! - the tracky bottoms, the "decorating clothes", the badly stained but keep just in case...

Now put the smart things away. Put the Not Outside! things in ONE drawer (you don't need more than a drawerful) and keep for either a slob day or a severe housework day.

The nice clothes, that's what you wear. Only buy things that will look good on you when you go out from now on.

Personally, I do not like tracky bottoms at all and won't wear them. I wear sports dresses or shorts/t-shirt when exercising. I like looking nice. I have clothes in sizes 8-12 and was able to wear normal clothes from the off - yes OK so size 12 skirts that sat below where the bump was with a longer cardi, but I looked nice! Blouses over feeding singlets make you look really nice and they're SOOOO easy as they keep your arms warm too. Yes, you unbutton to feed but the singlet's there so you don't ever reveal anything.