You need to let people know that he isn't happy being passed around & that you would rather he wasn't picked up, held etc by others.
I was worried when I went on a holiday to visit the IL's who the kids hardly see (because of distance) that they would be offended by the boys not running to them with open arms. But thankfully they understood the fore warning I gave them about not expecting the boys to realise that these people are related & therefor instantly best friends.