Are they not just asking out of polite interest/making conversation? As with others, my first thought would be that they were looking for recommendations.
You seem, perhaps, a bit self conscious about your own reasons for not using it. Or am I reading too much into it?
anyway, if people have particular ideas they will regardless of what you do or don't say.

I've used daycare out of need and for my own benefit to get a break, and I'm quite comfortable with that. I totally appreciate other people's reasons for not using it at all and I don't feel judged if someone thinks daycare is inappropriate for young children. Actually, I agree that it's a compromise. But, as with many of my parenting decisions, I am less than perfect and sometimes the compromises are quite deliberate. I'm ok with that.