I wouldn't worry at all - I cannot have any milk in my tea when pregnant - I know the taste you are talking about it is foul! And milk always smells off to me when pregnant too.

I think it is just your preg playing tricks on your olfactory