I hate to say, but I think there's something to it as well. I will say though, that some are more sensitive to it than others.

The first two times we got a BFP were when I had given up sugar (and dairy). I have endometriosis and part of the endo diet is to avoid sugar (along with dairy, gluten, red meat, soy, anything artificial, etc). I'm trying to remember, but I think it can cause inflammation in those sensitive to it. I was eating sugar when we got the 3rd BFP, but only in small quantities.

I understand your struggle. I know I need to give up the sugar, but I'm struggling with a major addiction to it at the moment for similar reasons (lots of emotional stress). For now, I'll be happy if I just cut back. I am switching to dark chocolate (in addition to it containing less sugar, I find I eat smaller serves) and cutting back on soft drink. If I can make those changes stick then I'll work to improve them further.

One thing that's really nice for dessert is some fresh fruit with a good quality greek yoghurt. The yoghurt I buy is sweetened with honey, so it does have some "sugar", but it's much less than conventional desserts. If you don't mind the tanginess, unsweeetened yoghurt is lovely with fruit too.

*edit*
DD was from IVF#5 and our second BFP (ie sugar-free).