In general usage, yes we most often consider that indigenous refers to aborigine people. But, the correct meaning of those words is as I described above.
I am guessing that is where the offense stems from- by not correctly referring to someone, which can be seen as a lack of respect.
And if someone is offended by that, it is therefore offensive to them and not for us to say that it is not offensive. (That sounds really weird, but you get what I mean)