The thing with caffeine is that it is out of your system in a few hours but takes a day or more for a baby to process - although they get a much tinier amount.
So the timing isn't usually an indicator if that is the problem.....