You know, Some people are thin and some are not. I gain weight just by looking at food, my DH has to eat junk food non stop to gain weight.
My DD takes after her dad, she has never been a cubby baby, i have always seen her ribs and spine. She was BF on demand and was always in the 25%ile when she was weighed (although she was born at 3.8kg)

I would be inclined to say that if you baby is not looking for food, giving him extra is not going to benefit him.

Try not to worry, babies are exceptionally capable of letting us know when there is something wrong!