Oh puppies, it seems this is more of a question of your relationship than libido. You are doing excellantly by being able to give him sex 2 or 3 times a week. There is certainly nothing wrong with your libido. The "maybe affair" will have more affect on your sex drive than the pill, I can promise you. And at the same time, if by some reason you were to split, it would be a bit unfair that he is unable to have children with anyone else because of the vasectomy. I think that you would have to work at your relationship rather and he should rather admit he might need to change his sexual thinking of needing it all the time. There is nothing wrong with you and if you really don't want another child with him, then don't go off the pill. I hope things improve for you soon.