There's a lot of factors I think. I think eggs can be pretty particular as to which sperm they let in.. it's not always first in first served LOL. And even if the egg is fertlised, it still has to make the journey down and implant in the uterus, and that's not a certainty either. Then there's the environment within the uterus, can be affected by foods, illness etc, which can make it sperm friendly or non-friendly, and can possibly affect implantation too.

If only it were as easy as sperm meets egg

Good luck with your TTC. Remember tho that even with younger healthy couples it can take 6 months or more of trying to fall pregnant. So don't stress just yet