Was he aggressive when he was with just you and DH? I agree with Kazbah, cats are very different when they are around other cats. My cat won't have abar of other cats, but despite years of harrassment at the hands of my now 4 and 2 year olds, has never even swiped them. I would see if he calms down back with you think about getting rid of him.