I wouldn't worry about swapping from public primary to private high from the kids' perspective. It can be a good thing.

Apparently, there's research to indicate that children in the same environment for all their schooling struggle at university as they're not used to dealing with different dynamics whereas those who dealt with a change of school take it more in their stride. From a personal perspective, I can certainly vouch for that. I was with the same group of kids throughout my schooling and I think, in retrospect, as nice as it was it actually did me a disservice.