I think it's lovely that he feels comfortable going to the neighbours house. It's probably not so good that he disregarded what you asked him to do (or not do). It's a tough one.