I don't live my life not giving friends the respect they deserve. Maybe your friends don't think it rude for you not to rsvp. Mine do.
I seem to have an unspoken agreement with my friends that silence should not be taken in an offensive way. That works fine for us. No one feels a loss of respect.