My firstborn was 4.2 kgs, so I have copped this with all of my pregnancies since. I have had many heated debates and many tears after being told some horror stories about my apparently gonna-be-huge babies. Out of the babies that have been born at term, my last was the closest to my first borns weight at 4.1kgs. My family have bigger babies. My nanna had babies that were all 9lb & above. My sisters last bub was 5.1kgs. Neither had GD.
I feel your frustration. It would be SO nice to not be dumped in the 'might have had GD' basket.