When I was trying for a baby (7 years it took!) I used to wonder about the downsides too. I remember hearing some academic talk about how parents only say they are happy because they have invested so much into parenting. He compared it to buying expensive socks then realising they are just socks but raving on to your friends about how great they are because you spent so much on them!!! Hearing that scared me! However now I am the parent I would say he is wrong. Yes it is exhausting and your life changes forever but that is it, your life changes along with your attitude to things. Eventually the person you were is in the past and you are now a new person with a child and it is hard to remember how you thought about things in the past. The same way that when you were at school you couldn't imagine working for a living and then suddenly you are that grown up working and paying a mortgage or whatever and it is just life. And you get on with it without really thinking about it.
In regards to the sock analogy, your idea of happiness changes too. I now think a cup of tea is heaven and a fifteen minute sleep in, while DH gets DS ready, is absolute bliss! Noone could have told me that would happen prebaby!
This is without even mentioning the love I have for my son. When they are here you just cannot imagine life without them. It is hard to imagine when they are an abstract. I know because I spent 7 years trying to imagine the abstract baby!
I remember you asked a question earlier about financing for a baby. Maybe you are overthinking this because you are not ready?? Although I remember a friend saying to me that you will never be 100% mentally ready for a baby but you need to be at least over 50%!