I would never cut down to low fat milk. The milk you buy sounds perfect.

I would look at reducing/eliminating refined and processed carbs.