I don't think it is. Which is why I use neither word, and if someone says it to you and you don't like it then I would definitely say something about it.

The reason I think it's ok is because generally when I grew up you strived to be skinny it was a positive thing. Where as being fat was a negative kwim?

I think now thanks to extremely underweight celebrities skinny has also become a negative.