That is not fair for your partner to be making you feel so rubbish. I think you have to do what is best for you, it sounds like you have been very understanding of his trouble with his other children but you have your life to think about too...