I don't think you are doing the wrong thing, but could you or DH be waking him up when you guys move around/make noises? Maybe it would help to move him a little further away from your bed.