I just couldn't imagine carting my poor darling the 40 min drive to his house, then not get home till well past his bedtime, what about his bath? where will he sleep there? (if he sleeps at all there) if we wake him to take him home, will he settle as normal into bed once we get home? will he get upset and cranky and tired at my dads?
Something that works for us when we have dinner at the ILs is we bathe the kids there, give them their bottles and generally they fall asleep in the car on the way home. An upside to this is that the MIL will generally bathe them and its something nice for her to do with them.