check whether there are preservatives in the bread, if the same one is in that then maybe thats not it, though some intolerances and allergies are cumulative, so a little bit may not cause a reaction where a lot can?
DS gets contact rashes (the redness you are talking about) from tomato based products.