No I don't think it is selfish. I think it depends to what you want to be 'rich' in. If you want to be rich in possessions and labels and the like you probably should have less children, but if you want to be rich in love and family relationships and support then a larger family is a great option for you!
To me the things you would take away by having more children (possessions) are the things that aren't all that important to me, and the things you add (siblings to love and face life with, the support of a large family) are way more important.
Having said that I do realise that not everyone has the luxury of those options. I wouldn't say we are lucky (because it isn't luck, it is through hard work and smart decisions) but DH and I both have uni qualifications and had well paying careers so we have different options to what others have.