I'm sure people only have our best interests at heart when they tell us to "give ourselves a break", but yes, having to constantly explain why I feel the need to express rather than formula feed can get tiring!
It did my head in for a bit. In the end, I didn't explain to people. When they asked why, I said "because I want to". When they asked "how long are you going to do THAT for" I just smiled and said "until I'm ready to stop".
It's about you, your baby, your partner. Everyone else can have their opinions, but it doesn't matter; you are doing what you need to, and you're doing a great job

I understand the sadness too. I still feel it... not as bad now.