As long as you and your partner are happy, who cares what anyone else thinks. I know its hard, but most of the time they have no idea what its like for you guys. Its a solution to a problem that you both are okay with.

Ive lived in a country town for a couple of years when I was in school and I do now - it is the worst part of it, gossip and people wanting to know your business. Less boundaries and personal space.