I assume someone will be walking him to the bus stop and picking him up from there in the afternoon? I do think he's too young to get himself to the bus stop! But if someone is putting him on the bus and collecting him, I think he will be fine. I would suggest you make friendly with the bus driver and give him your mobile number in case there is a problem (you are running late or the bus is!). One of the mother's from my daughter's class has had her daughter on the bus since prep, she takes the driver a Coke every afternoon when she picks her DD up from the stop! Also, see if one (or a couple!) of the older kids would agree to keeping an eye on him for you - not being responsible for his care, but just making sure he is ok and knows what he's doing.

Honestly, I don't think it's ideal (nor do I think you are an ogre!), but if it is the only way for your DS to attend your preferred school I think you can make it work. Put some back-ups in place, make sure he feels safe and secure and knows exactly what to do and he should be fine. Good luck!