Doesn't even sound like he knows what he's supposed to be 'learning' How to not get caught, perhaps? He needs to take a much harder look at himself and to why this happened in the first place. He needs to talk to you about it because you are the biggest stakeholder.
You guys need a third party. When you started this thread you wanted to believe him, you wanted to know what to think from other people. Now you're acting on what's inside you and it's bringing up important things - you guys need to navigate this together if you're not inclined to give him the boot and if he's inclined to stick around.
You don't deserve this, by a long shot. Get angry if you need to, cry if you need to, let him have it if you need to.
Do what you need to do xxx