Go with your gut instinct honey. One of my twins a week or so before he turned 1 was very sick with a viral infection (which tests tell us was the influenzae (a) virus strain), he also copped a bacterial infection on top of the viral and refused to drink milk. We gave him hydralite powder instead. Lucky for us he liked it but the doctors were not concerned about him drinking milk. Said to us that as long as he takes any sort of fluids, we were all good.

Personally I wouldn't stop giving my child milk if they aren't refusing it. But that's just me.