I don't think it's an allergy sign, but it could be a sign of intolerance (allergy signs usually show up straight away). DS had a dairy allergy which he grew out of at about 13months. He would get a red rash on his face, starting around his mouth, within minutes of eating yoghurt.

It would be worth asking GP about dairy intolerance although I think there would be stomach problems as well if that were the case.

Good luck.