I don't think trying to improve your own health or your your DH's at any time in the TTC journey is a bad idea no matter what the cause of your fertility issues is.
Just to let you know of an ICSI success story we had 7 stim cycles in total and from the 2nd cycle onwards all the eggs were fertilised using ICSI. Despite my husband having very good sperm on the initial testing for some reason when left to their own devices two sperm were managing to get into my eggs (super sperm? more likely not so strong eggs!). So there are many reasons for using ICSI to control this step in the process and it is not always just for those people who have only a small chance of IVF success.

Try not to read too much into all the diagnoses you are given and stick with the things that you can control. Your own health, exercise and attitude and if you decide to do IVF then the support is out there to get you through it.

Good luck