I would have done (and have done) the exact same thing.
My girls are VERY adventurous, they do things most girls their age would not even contemplate, they follow the big kids, they ride motorbikes, the jump on the trampoline and go on the 'big kids' stuff at play centres.
We have slowly realised they will let you know when they are not comfortable and more often than not the bigger kids have the best of intentions.
My girls were constantly on the trampoline at family parties with the bigger kids, at times the bounce or fall was a little hard but nothing that 'injured' them.

Maybe you can chat about it with him, ask him what made him uncomfortable and what makes you think you didn't care, i'll send you some pics of things my DD's weer doing at that age, he'll soon realise he has nothing to worry about.

Tell him my DD climbed UP A LADDER to our roof line before she turned 2