My ds#1 looks like my dad, too a tee, but my dh is not his dad. my 2nd and 3rd child look like dh's side of the family. but ds#1 gets told that he looks just like dh.. people have their own perceptions of children looking like parents/grandparents shoudl look like (does that make sense??) So i wouldnt worry about it, they could grow up and look just like you, and nothing like dh or il's.

HTH