Is everything looking good right now? If so, your'e probably ok leaving it so longas as you're still getting lots of poos and wees and weight gains. It is quite likely that he'll 'wake' a bit more in the coming weeks so things may change on their own. That's what my DD did
The unsettledness in the afternoon is normal of course, and the cluster feeding all counts towards total feeds and helps him to go longer overnight. That's your payoff there so I wouldn't discourage it