I don't know if it's rude so to speak but I would consider how your guest will feel. Even if you say no gifts they will feel obligated to bring you one (I'd like to see a poll on how many people still bring a gift even if the invite says not to iykwim). I think people will give you gifts if they want to give you a gift regardless of having a baby shower. So if it's the gifts that you want maybe save yourself the hassle of the party and the potential of people thinking you are rude.

I don't think we will have a baby shower because we don't want gifts! We don't have much baby stuff but don't want people buying us stuff left right and center just because they were invited to a 'celebration'

If it's really not about the gifts for you than why not just have a party with your family & friends both male and female and leave the word baby out of the invite all together. If I received that invite I would just think that you and your dh were throwing a party just because you want to let your hair down as soon you will be too busy to see anyone.