It's probably a bit of both.

And I've thought similar things as you, about the universe punishing me, but it was more like I'd failed the test with DD and wasn't a good enough parent to deserve another. Haven't thought that in a while now (thankfully) - I've seen too many "bad" parents (eg drug addicted mother who caused her child to be born severely mentally and physically disabled and then let him almost starve to death as an infant before he was taken away from her at 4mo and just had another child this week - #6 for her - which was taken away from her immediately this time) have child after child to believe that I'm any less "deserving", so my focus is more on the issues that are causing our IF - ie endo, DOR (caused by the endo) and MFI. Which means, rather than feeling "punished", I mostly just feel angry (about all the Drs who dismissed my pain, leaving my endo undiagnosed and running rampant for 20 years to cause my share of our IF problems)...