I'm not 100% sure, but from what I remember, smoking and drinking especially do affect fertility, especially in men. I'll do a bit of googling though because it could be a wives tale, you know? But from what I remember it does. I too was ttc for 2 years and only had a handful of drinks, but my DH stopped drinking with me