No, you're not. He needs to man up to this. I'm sorry, he did this & he has no right to down play your feelings. He's guilty, but doesn't like the feeling it gives him. So he's trying to justify it to himself, not just you.

Turn the tables on him. Ask how he'd feel if you'd done it to him.