What a fob off!
The IVF thing is completely irrelevant, unless there's some underlying issue that may affect pregnancy or birth.

For the meds, I don't know, but surely a GP (and yes, perhaps a different one) can handle that side of thigs while a midwife handles the pregnancy stuff (communicating between the obviously to ensure seamless care of you).

ETA - it's different, but my GP had a similar response for me. Because we went with midwifery care 2nd time, she wanted me to see the specialist to manage my thyroid stuff since I wasn't seeing a doc for the preganncy. paid $150 to see the endo who said "Your gp can handle this".