UPDATE:
First thanks to everyone again.

I spoke to him just then and told him that he can't stay here. I looked him in the eye and said "You are leaving me for another woman. It's not respectful that you continue to stay here". He just nodded and said he'd "find a room". Whatever. I said I don't care if you stay there, I just don't want you here. There was no hysterics. I'm done crying.