Nope not rude.
I too always take my shoes off when I get to someone's house (that is if I'm wearing shoes!) even if they tell me it's fine to leave them on... I think it's rude to leave them on.
We have just moved into a rental property and everyone takes their shoes off when they visit, except for the in-laws!! What p***es me off about that is you aren't allowed to wear shoes in their house... What the!?!?!