From my POV - you are doing what is right by Matthew is letting you guys have your cuddle time in the morning. Let Matthew be your guide, if he enjoys this sleep time, then let him have it I say.