I'd hold him off with some healthy snacks around 5-5.30pm (piece of fruit, some vegie sticks, or similar), do some of the bath time stuff or other chores before DH gets home, and then sit down with him together to eat your meals. It's no wonder he gets distracted - he's the only one eating, there's heaps more fun things to do and you're probably there to keep him company more than enjoy the meal with him atm.

It's a tricky thing to juggle when they're hungry with family meal times, but four should be old enough to try to meet somewhere in the middle.