hun at the end of the day you do what is right for you
FWIW my twins were never left with anyone - not for one sec besides my DH and I until their first day of kinder
they were fine!
but age 3 they were ready to be left- they understood it all and there were no tears

they are very clingy kids and tell me how much they miss me everyday after school but they understand that they need to go to school and they feel safe and secure at school so it is fine
i really dont think that you need to get kids used to being left in time for school- the best thing to get them ready is TIME and them growing into that stage
its all up to you- if you want to continue to do this then go for it- but if the price of you both being not happy is not worth it- then i would wait for a bit and maybe try it when he is a bit older
hugs hun and just do what is right for you!!!!