It's not a silly question cause I asked the same thing when we first saw our fertility specialist as my dh is in his eyes not well endowed, I personably think he is fine.
I asked the question to our fs and was told size has nothing to do with it. This question was much to the embarrassment of my dh but I had to ask!!!
So just go for it and enjoy the trying to conceive :-)
I can sympathize with your Dhs embarrassment. My DH asked my Ob (who I work with as I'm a midwife for over 7yrs) if Oral sex caused my ectopic pregnancy. I've never been so embarrassed in all my life. And thought my husband was stupid also lol