I'd talk to him about it. Maybe talk about how you & your DS miss him or something like that. Or maybe talk about how important it is to spend time together and ask if he has any ideas about how you could do that...? Then it becomes his idea.
DH makes a point of leaving early for this very reason.