I agree with everyone else's opinions - I can see that it would be a hard conversation to have because you would already be feeling so hurt by his actions - I can see it blowing up when you want to talk about it (but I guess that just means I probably would blow up about it lol not you sorry)

Maybe another option would be to suggest that he focuses on leaving in the morning and you could all go for a walk together on the afternoon? That way he has daddy time too.

Maybe bringing it up as you being worried that he isn't getting enough daddy time as the problem so that he can come up with the solutions ;-) that always works well for me lol

Good luck with it - I hope you can sort it out without too much trouble.