Depends on the occasion. I hate going to bars for example and not being able to have a glass. Pretty much the only reason people are there is to mix alcohol with socializing.

House party on the other hand, or BBQ or something, I am much more willing to abstain happily. Although yes people do assume you are pg.

But I don't think it's rude not to drink, no! Personal choice and all that.