Its another one of those things where some people say yes some say no. You have to work out what you feel comfortable with.

After 4 years TTC I say no to caffeine, as well as alcohol and a few other things but I'm comfortable with that.