I agree it says more about them then it does about you. I also think some woman believe you need a man to be happy so if you don't have one then you must be after theirs

Take it as a compliment and don't take on anything else, you don't have to justify yourself to anyone else.