I think most babies go through something similar at one stage or another, and with some kids it's just part of their personality. My 19m old is a sensitive little thing, it takes a while for him to warm to new people & situations and he let us know this from a very early age (10-12w). I let him take things at his own pace. It's totally okay for you to let people know that your bub's not comfortable with being held by other people at the moment and that it's best to approach him slowly.

All the best.