I am almost 12 months down the track of TTC after the loss of my Caitlyn and still yet to get pregnant (and yes, on the drugs too although without a confirmation of PCOS. Just poor ovulation ) I completely understand the desire to TTC so soon and we have already made the decision to start as soon as we safely deliver one (assuming we EVER get pregnant).

While their fears for you and your family may be valid, the choice and the your ability to cope can (and should) only be decided by you and your DH. I wish you every success in whatever decision you make.