My endocrinologist, who is a top specialist in treating PCOS and related things (one of the Monash Medical Centre research team), told me to stop taking Metformin as soon as I knew I was pregnant. So I stopped taking it at about 4 or 5 weeks. Now at 19 weeks, so all is well so far.

My impression from what she said was that the effects of such drugs on a foetus have not been well enough studied to know that it's safe. She knew how desperately I want this baby, and didn't want me to take any risks.

Probably the risk/benefit balance would be different for different people.