I've got 2 kittycats and haven't had a problem. Since I found out I am pregnant DP changes the litter tray. Both are indoor cats so our litter tray needs to be changed everyday. Occassionally I have scooped it and have just worn gloves and washed my hands afterwards. I think your docs opinion of getting rid of your cats is a bit extreme...

My cats will be banned from bubs room but we will still have a net over the cot. My cats are sooky la la's and will probably hide from bub anyway lol