You cannot put your life on hold whenever he wants you to. If you have made plans stick to them. Every weekend you cannot be sitting at home wondering if he is going to turn up. You have told him you need notice if he is coming so stick to your decision.
If he fades out of your life later on you will still have a routine to follow and not waiting 'in case he turns up'