I bought my first house at 24 and have sold and bought twice since then. However I waited until I was in my 30's to have children - both times got pregnant first attempt (literally) so it's not a given that you will have problems concieving later on.

I think the only thing is that it is easier to save more money when you don't have children than when you do so if you are looking to TTC before buying I would still start saving for a house right now and putting away as much as possible while you can.