Pfft. I have had some of the best times with sober friends whilst I'm sober or drinking. I think so long as the sober person doesn't make the drinkers seem odd for doing so (which I doubt you would!) then don't worry. I think your hubby worried too much