I think if you and DH are swearing, even if only occasionally, then you aren't going to have any luck stopping him without making it a family commitment.

Monkey see, monkey do, and if he's doing it out of frustration, then it's at a moment where he isn't going to stop and think, it's only something mummy and daddy are allowed to say. If you are all trying to stop together, you'll probably also appreciate just how hard it is for him to break the habit as well.