I think they will always sag whether you breastfeed or not (because of gravity's effects), but it will take longer if you don't get pregnant or breastfeed, for the reasons others have suggested.