Went to the doctor today and he told me that it's a nerve. Nothing that can be done about it, everything should return to normal after the baby is born. He said he sees this problem in about 1 in 50 pregnancies.
Oh well, that's the only problem I have so I can consider myself lucky.