I think too when people say you look small they're just trying to be nice in telling you that you don't look like an enormous whale! I would try to take it as a compliment!
But it's true - pregnancy is like having an idiot magnet on your head - so many people feel compelled to say something stupid to your face and think nothing more of it.
Pff.
It could well be if you are indeed measuring small that it is genetic - small father could indeed be the reason for a small baby - but that's no indication that there's any problem.
I'd be more worried if your husband or partner was a giant... hello, where's my pain relief.

I get conflicting comments all day...
"you sure there's only one in there?" is followed by "gee you're so tiny!" is followed by "still 9 weeks to go? Are you joking?" is followed by "you're very petite".