Of course you are doing the right thing. Unless thetes a danger in them being with him, in which case you obviously wouldn't be giving him access. It good that he's seeing them before the weekend too so he's not so much of a stranger iykwim?

I think it's a really good plan.