stuff what everyone else says, if u want to keep them why shouldnt you?? kids and pets can live in perfect harmony, so long as the cats dont have access to bubs room there wont be an issue at all.
next time someone brings it up just say, whats it to you?