How do you get on cooking for your family when your baby needs high fat foods and the adults need low fat foods?

We are not giving our son dairy products just yet, but when we do he will need full-fat milk. If I make a meal containing milk should I make his one separate (seems like a hassle), or do my husband and I just have to deal with a higher fat diet?