DH and I have had this discussion many times.
The people that he knows that went to a small school for primary and then mainstream high school were really well balanced and confident people.
But my brother went to a small rural school, about 20 kids, and he got passed over. Not intentionally, but he memorized all the books on the first read and then repeated them by rote. By the time anyone realized that he wasn't actually reading he was nearly 9 and it took a lot of extra tutoring to catch him up. I attribute a lot of the problem to the teacher being spread between all year levels, so if a kid seems to be doing all right they got skipped for one that needed more help iykwim. But I guess that happens no matter what the size of the school.

There are two small schools with only a dozen children at each around here, but there is a teacher at MG that does a day a week at each of them and it seems that a lot of the kids that are going there that have behavioral issues. I would make sure I spent a decent amount of time there before enrolling to check out what the other kids were like because the smaller class size can be attractive for more challenging kids - you don't want them to be stuck with a Tanya Calkman.