I wouldn't leave him there. Your baby deserves to be happy and so do you and it sounds like neither of you are under these conditions.

Is there anyone else who can look afetr him while you are working or would you consider child care??