I had GD (diet controlled) with DS1 - he was 2.9kg at full term.
I didn't have GD with DS2 and he was 3.6kg full term, nor with DS3 who was 4.5kg at two weeks overdue.

It's DEFINITELY possible to have an average-size (or bit less than average) baby when you've got GD.

PS: I birthed my 4.5kg son naturally without any dramas whatsoever