I have a cat, I cleaned her kitty litter while pregnant with all three, just used rubber gloves. She is a long haired cat and I have to brush her everyday too, so you should be fine, just remember to wash your hands is about all I would say.
My OB asked if I had a cat and I think one of tests done was to check for Toxo and came back fine.