It's certainly up to you as to whether you go ahead and try!
Statistically, yes, age does affect fertility. That's on average, across the population. There are obviously individual variations, as with all things.
I was just looking to ask a similar question. We are TTC #2 and have been for a couple of years - with illness for me, it has not been successful. I was 38 when I conceived DD and 39 when she was born. We had a number of miscarriages before she was born, but I believe they were due to a number of factors, not just age. I place hope in the fact that my mother conceived my sister at age 40 and my grandmother had her last child at age 47 - hoping the reproductive genes have gone my way.