Like someone else said caffeine stays in your body for at least 24-48 hours, the exact time depends on how your body metabolises it.
So if it effects your babe, then there is no good time to have it. But fingers crossed it isn't an issue for you and you don't have to worry about it!

I have a 'real' coffee every morning and we don;t seem to have any issues here.