Yep I'd say start the switch to water. DD only gets water overnight now and I've got a sippy cup that doesn't spill, so I often just give it to her and let her keep it in her bed.

She was waking and asking for milk... So we started telling her that milk was just for during the day and before bed, not overnight and that at night she would only get water. We just persisted and she eventually stopped asking!

Good luck...