Maybe just not tell him anything about the birth from now on. That part is about you and your body. I would just deflect any questions and not enter into conversation with him about it (i do it with people in my own life who just don't need to know details).

I know breastfeeding does impact on visitation, especially in the first 12 months, but don't know specifics.

ETA Australian Breastfeeding Association - Continuing Breastfeeding after Separation & Divorce (thanks NTL)