Maybe he's just tired? Could you try feeding him a bit earlier?
I don't think he will starve. They usually take what they need.