Be true to yourself, just no. YOu don't have to justify yourself or even give an excuse. Just Say NO!
I wouldn't be offended, but I wouldn't mind knowing why, either... if you just said no, or that you don't 'do' them it might leave me wondering what the problem is (asusming that if theoretical 'I' am holding the party, I don't have a problem with it)
I wouldn't think you have to go into great detail, just a basic ... '_' .... erm, I don't know what.. just so they know it's not personal?