Don't worry, mate! People will say stuff regardless... You will have people telling you how MASSIVE your bump is one day, and the next you'll be told you are very compact and small for dates. What can you do?
Personally, I put on HEAPS of weight in the 1st tri because I am shocking on the fatty food and carbs when I have m/s. (Feels kind of like a hang-over.) I have already put on 10kg this pregnancy and I'm only half-way. But I can pretty much guarantee you that most of that is just body fat from being such a guts and not doing any excersise.