From what you have said I think the amount of time your son spends aroud other children is fine, If he plays well around other children but enjoys his own company at home then thats all that matters.

I feel little ones need to be around other kids to learn how to interact but if they are only children they are used to being by them selves, My eldest was an only child for twelve years, she is now a well adjusted ( Very social )teen she also had cotact with cosins and a few friends kids when she was small.