I can totally understand where you are both you and your DH are coming from. I suppose our reasoning for stopping at three kids is also a financial one. Adding another child, would mean I would be a SAHM for even longer and would set us back again as far as being 'comfortable' and buying our dream home etc etc.......(and I am starting to enjoy my independence again. LOL)
I'm sure whatever you decide it will be what is best for you and your family.
Most importantly take time to look after yourself aswell.