I would talk to your OB or MW next time you see them.
Not sure what State you are in..my DS2 had a dairy intolerance/allergy from a very young age, he was 8mths when we had him tested. We got him into and Paediatric Allergy Specialist. She was fantastic. They would do the scratch test on the back.
RCH in Melbourne also have a great allergy clinic.