i let DD stay at XPs since she was 3 months old. And yes i agree the older she gets the harder she is to leave cause she needs amusing more and gets seperation anxiety from me (whereas at 3-4 months she didnt care who had her).
if your DS has a routine id write it up along with list of important/emergency numbers to give to the caregiver....thats what i did for DD and gave it to XP and it makes me feel at ease. HTH