I think that too, but when i hear the name and not see the spelling i think girl straight away?!
You wouldn't hear it as a middle name. Not much, anyway, and you'd have the first name first to make it clear. As long as the first name's not Ashley, I think you're OK