I don't think it's rude at all...actually I think it's rude for people NOT to take their shoes off when they visit other people's houses, unless the host says don't worry about it... I was brought up with not being allowed to wear shoes inside so maybe it's just because of that...I wear shoes occasionally inside but mostly leave them in the spare room at the front of the house.
I would never just walk into someone else's house with my shoes still on!