My XH did this a lot with me. So I gave him notice that unless he tells me the exact time he's going to be there, at least 24 hours in advance, I assume he's not going to be there and make other plans.

If he is late for his pick-up time by more than an hour, I leave.

I too didn't want to 'stir the pot' but I soon realised that he was using his visitation time to jerk me around. I was nice as pie about it, I really was, but the msg got through - I wasn't going to be waiting around while he came and went as he pleased.

The initial discussion was a bit tough, but it's been years now and he hasn't fooled around for a long time when it comes to picking up/dropping off on time, because he knows I won't put everything on hold for him. And as a result, XH and I have a better relationship for it.