I put on nothing to 17wks and now at 28wks I have lost 3kgs. But that is due to being diagnosed with gestational diabetes (so avoiding all junk and processed foods), and not having much of an appetite. Dr's aren't concerned. Bub is still growing well, measuring a week ahead and moving around lots.
Don't worry about other people's comments. Everyone is different. As long as you are feeling happy in yourself and your Dr is happy, try not to let the comments bother you.