I drank a large iced coffee every day during my first pregnancy and it hasn't appeared to have had any adverse effects on my little one.
I think the reason that they warn against caffeine is that research has shown where you are at high risk of miscarriage even a single cup can contribute. If you don't have any real medical reasons to be concerned about miscarriage (like family history) a few cups a day will be okay.
I think it also depends on what your body was used to before you fell pregnant.