I think genetics plays a part but only up to a certain point.
My mother's family have always been very fertile (not one having trouble) but my Dad's family have had all the trouble.
My DH has issues as does his sister.
I am the only infertile one in my family but I am fertile its my tubes that stop me getting pregnant so I can't blame that on genetics but environmental factors.
A friend of mine hasn't been able to concieve (PCOS) all her sisters (3 of them) are exactly the same. One has concieved and had a DS and the others haven't been able to.