It comes down to your being at your healthiest, IMO. If that coffee each day makes you feel great then I reckon go for it. If drinking coffee has a negative effect on your body (ie makes you nervy or keeps you awake at night or buggers up your digestion) then maybe it's not a good thing for you to have.
I have quite happily drunk at least 1 coffee per day whilst TTC, pregnant and breastfeeding. That's my groove and you'll find yours. Best thing though is not to stress cause I'll wager stressing over it is worse for you than the caffeine.