Off the top of my head....Toxo is spread by contact with the faeces of the cat. If you cat has toxo then you probably have immunity to it already. Enlist someone else to do the kitty litter or wear gloves while doing it. Wash your hands well afterwards and before eating.
Cat scratch fever is not that common and as far as I am aware no more of a risk during pregnancy. It is also usually quite benign and goes away by itself.

I vet nursed throughout my pregnancy including cleaning litter trays (with gloves) as I was likely to have been exposed and I did what I could to reduce my risk. My guess is she has never owned a cat. Getting rid of your cat is quite extreme unless you are also immunocompromised or the like.

I would get another opinion if you are worried. Perhaps your cats vet can give you another opinion, especially if they are a female with children.
Hope this helps.