You're definitely not crazy. I have declined it this pregnancy. I just don't see what the benefit is of it. I believe I would 'know' if I had it, I would feel it within myself, and like you say diet and excercise has a lot to do with it. So if I suspected I had it I would address those issues first and go from there.
From what I have read of your posts you seem to be very in tune with your body. Go with your gut feeling, if you don't need the test don't have it.

There are always going to be people who believe you should have it and people who believe you don't need to. If you haven't already I would highly recommend reading some of Dr. Sarah J. Buckley's stuff, she has some great info about this!