At this stage you have a pretty well established milk supply - it shouldn't make much difference. I drank a couple of cups a day with no noticeable effect. If you see a change then you can always stop - your supply will fix itself.