Maybe show him this thread? Show him that your reaction is actually fairly lenient, compared to what other people would do given a similar situation. But also to show him what your real feeling are. I also think it might be a good thing if you lost it with him. Show him that you really have taken this VERY seriously and that his actions are not nothing. This situation isn't to be treated lightly. This is your relationship and your feelings he's playing with.
And if you can, get away. Be alone. Really take the time to meditate over it all. Think for yourself, without anyone else (him/children/family/friends) being part of it.