I would trust your feelings. Having a day together as a fam is great. Would you mind if they had one child at their place for a couple of hours but wouldn't go out? I let my MIL have Coco (she minds her once a week) but I request they generally stay home. i am getting a bit more relaxed, at first I was a mess, but Im having to let go now particularly with an 8 year old.
Take your time and never do anything if it feels wrong. All the best xo