Does your cat go outside? I know that risks are significantly reduced if they stay inside all the time (i have two inside cats).
I would think as long as you are worming them and bathing them regularly there would be no increased risk. Make someone else do the litter as well so your not exposed any infection that way.
Cats are relatively clean animals, and i know they do probably carry alot of germs, but as long as you are vigilant about cleaning up after them and cleaning your house on a regular basis i dont see why you should stress