I know it seems like something to be worried about, but when your boobs don't get that 'full' feeling anymore, it is a wonderful sign because it means your supply is fully established and your baby and body are communicating perfectly so you're making exactly the right amount of milk for him!
I wouldn't stress about how your boobs are - I'd be looking at how your baby is. Is he happy and healthy and full of life? Is he his normal, boy-ish self? If he's fine, then your milk is fine.
Everything you've said, is the exact advice I would give to someone who is concerned about their supply. I think you are much wiser about breastfeeding than you give yourself credit for!! Perhaps the only thing you need to work on is your self-confidence, have more faith in yourself mate because you are doing an absolutely awesome job
I don't know much about the effect of IVF medications on milk supply but the ABA would definitely have some wisdom to pass on - 1800 MUM 2 MUM give them a call and if the counsellor you speak to isn't sure of the facts, she can definitely point you towards your local ABA group leader who will have a fact sheet about it for you.
Much love to you, I hope the next IVF cycle works for you
Bookmarks