I had the same thing happen to me - and what I was told is:
The calorie count is different to what a dieting person / pregnant person needs. So by eating the food required for a dieting person you are not getting enough needed to help support you during your pregnancy.

You would not have done any harm to your baby at all.!! Try not to stress hun.