Yep allergies can certainly build up over time/exposure to the allergen. So I dont think its umcommon for people to get allergies when adults as well.
I guess the only way you will know for sure is if you go to see an Allergist.... or if you eat Peanut Butter again and get a worse reaction (which I dont recommend you do!)