I would be uncomfortable if someone wore that in my house, regardless of age or gender. It's not about sexuality. It simply isn't appropriate.
So if it isn't about sexuality than how is it inappropriate?