Honestly, I would say that if this behaviour continues for a week I would start him on solids. That way you can be sure it's not just a funny little blip in his feeding pattern or that he is just not feeling well. (And you some time to get used to the idea!)

A week or so earlier isn't going to make any difference to the magic 6 month mark. The fact that he is interested in your food is also a good sign. My DS wasn't doing that at all, I think I could have waited a bit longer before starting him.

His primary source of nutrition will still be breastmilk for some time yet, despite starting solids, so he won't be missing out on anything. I think this is where you have to take their individuality into account.