I can't stand passive aggressive people, and I take great pleasure in calling them on it. They won't say things clearly, just infer it. They want you to get defensive so they can sit back and watch you defend yourself and lead the conversation either where they want it to go, usually away from their own idiocy.

It can be hard dealing with them without buying into the game though - SO when you have to talk to him and he starts up, completely ignore the comment and keep the conversation on track yourself. Its also handy to visualize him with a penis on his forehead while you do this.

Then come in here and vent about the childish manner in which he carries on and thank heavens he has a new GF to distract him away from you.