DS used to do this too. It started out gentle and got harder and harder to the point I thought it must be doing damage. lol
Nothing I did would stop him and like you said he'd pull off the boob and get upset if I tried, so I left it and he stopped after a little while. Not sure how long - a few weeks maybe?