There are downsides to anything I guess. Being a parent is one of the most rewarding things you can do, hands down...the good far, far outweighs the bad.

What I missed was the lack of 'me' time, I felt the weight of the total responsibility - the fact that you are thinking about another person All. The. Time. There is no moment of switching off from that, ever, not even as they get older. I worry about them all the time, how they are now, how they will be in the future.
And I still have a secret wish, that one day DH will take all four of them away for a couple of days & I will be left alone.... but then, I would miss them all.

That's the real downside. That they are tied to your heart forever. And it's also what makes it the best thing you will ever do.