Does he open mouth breath or snore? If so, they'd possibly be better removed. This means they are causing trouble.
DD1 went in to have her grommits done & I had them look at her tonsils too as a gp had mentioned they were big. The ENT decided since she'd always been a loud open mouth breather & snored it'd be better to do it while doing the grommits. He also did the adenoids too. He had no medical reason at the time, but when they came out they were also enlarged - he said very enlarged, so it was worth having them out in the end.
DD1 also had more than 6 sore throats (not tonsillitis) a year, so that helped with our decision.