I'll answer it...

Firstly, you are very brave to push on in a relationship like that. I don't think I could do it. Sounds like sex is VERY important to your man and seeking outlets elsewhere, with other people instead of working it out with you, is not good.

But if you definitely would like to work it out, check out THIS thread. It will help address the questions and problems you have in a way most psychologists don't approach things, yet works so well.