Like many of the others have already said, you can certainly refuse to take the test, but I'm another who would always have it just in case. It's not the most pleasant, but I'm sure having undiagnosed GD would be much worse.
I've done it 3 times now and the easiest time was when the drink was refrigerated. It was so much easier to drink when it was cold. Perhaps before you decide where to have the test done you can ring a few of the closest collection centres and ask them if they keep their drinks refrigerated. If you rang in advance and asked one to pop a drink in the fridge for you I'm sure they'd be happy to do it. They have so many things for tests that go in fridges I'm sure they would have ample fridge space for one drink.

