I don't think it is selfish at all! It sounds like you and your DH have had a good chat about it and are happy with your decision, so go for it!
We do things for us, not anyone else. So don't let anyone try and tell you that you are doing the wrong thing.
Listen and take on board what you want but discard the rest