It is such a difficult decision, but I would follow my heart if I was you. As the others have said, we only get one chance at this and in some ways you get back what you put in. Our third has really changed our lives because I can't believe how busy we are but I don't regret our decision for a moment. I have heard people say once you have three, if you have more you don't notice the difference so much. Don't know whether that's true but it was a big jump from 2 to 3. It's a lot of fun though.
My grandmother ummed and ahhed and wrote lists with pros and cons about whether to have a third and ended up with twins so she got more than she bargained for! Beware the double whammie!

I think the bottom line is you have to do what is right for you and no-one else can tell you that. All the advice in the world will not change what's in your heart.