If affects your fertility because excess estrogen (from soy) can alter the ratio between your estrogen and progesterone levels and you need a perfect balance of both to sustain a pregnancy. The studies are out there and easy to find - go to google scholar and search for 'estrogen dominance fertility' and see how many hits you get, same goes for 'effects of soy on fertility'. There is an interesting report here http://www.litalee.com/documents/Est...20Problems.pdf and this passage in particular;

Dr. Peat’s research established that estrogen excess kills the embryo by suffocation (hypoxia). Progesterone protects the embryo by preventing hypoxia and by aiding the delivery of both oxygen and glucose. During the ninth week of pregnancy, estrogen increases. Women who lose their babies around the ninth or tenth week of pregnancy are either high estrogen dominant (e.g., hypothyroid) and/or are consuming commercial meat, poultry and dairy products containing synthetic estrogen (DES). Their progesterone levels are low and the liver capacity is insufficient to detoxify the estrogen. Studies indicate that progesterone also helps prevent toxemia and increases the intelligence and the health of the baby. How? By opposing estrogen, progesterone enlarges the cortex whereas estrogen actually decreases the size of the cortex and other parts of the brain.
Tasha, another alternative for you could be A2 milk. It contains a different protein to regular milk and can be tolerated by those who do have an intolerance to dairy. Other options would also be goat or sheep milk.