I'm booked into a midwifery led birth centre that is attached to a hospital and birthing suite. I've recently gone on AD's and now that I have I will probably be ineligible to birth at the birth centre as for them it is considered high risk, as the AD I'm on has risk of breathing/feeding issues. Every AD has risk to unborn babies so I'm not sure you would qualify for midwife only care. The IVF thing is absolute rubbish! And your GP sounds like she is putting you in the too hard basket - my GP has been nothing but supportive and if I have to birth at the regular birthing suite she will continue to do shared care with me.

Honestly I don't think it matters who manages your care and your birth, so long as you and your partner/doula will stand by how you wish to deliver, that's what really matters. Make it clear if you want to avoid intervention, no drugs, etc etc.