Like Trillian said, sometimes we all need big meals, and other times small.
Wait until he gets onto solids!!! Ha!

I used to stress when DD1 wouldn't eat, and then I got it through my head that there are times in my life that I'm not hungry, and I'd certainly get stroppy if DH forced it down my throat. So I learned to accept that this is a little person who knows when they are hungry and when they aren't, and they will let me know when they are hungry. (Believe me, she'd let me know.) I am so much more relaxed with DD2 about it.

He'll be fine, and you'll be fine. Tell DH to stop worrying and that everything sounds pretty normal.