I have no expertise but i knew a kid who spent a lot of time in Australia and Iraq (or Saudi Arabia, or both?? can't remember exactly), went to highschool etc in both countries, and in Australia, in school, we was a bit strange, didn't really fit in or appear to want to, was very hard to get to know.. He often talked about how different the countries are..
I think he was just a quiet, deep thinking person in general, but in my opinion the exposure to any culture that is SO different to Australia (Australia being so peaceful, with full government benefits, no poverty, democracy etc) makes it hard for kids to assimilate in either place. Or they grow up loving one and hating the other..
I also have twin cousins who live in Singapore and Australia half half every year, they are 10 years old and have made NO friends in school in Australia, and teachers have said they tend to isolate themselves.
But your DS is only young! I think it would only really matter at primary school age and then on.