If you have a daughter, wouldn't she think the same thing because you have big boobs and she has a flat chest? What if your son comes out with a bigger nose than either of you - that'll be different. What if your son has a birthmark on his face?

I doubt your son would think something is wrong with him (unless some older person puts it in those words) but he may notice it's "different" and ask questions (because that's what little'uns do) and it's a good opportunity to discuss those differences - in an age-appropriate manner obviously. If he's over about age 4 and is asking, then I'd say "when daddy was a baby, his parents thought the right thing to do was change the way his penis looks. But me and daddy thought when you were born you were absolutely perfect and nothing needed changing"
For an under-4 yr old, I'd say something like "yep, yours and daddy's penis's (penii?) are different because yours has a bit more skin - and look, your hair is brown and mummy's is red, and look, you have no freckles on your arms and daddy has some, and look, daddy's got a hairy bum and you haven't