My 2 cents worth - nope not crazy. If I were in your situation, I would have declined too. You are the best person to judge your unique risk factors and symtpoms. (Interesting, my hospital won't give me the option of declining....)
You always have the option of declining! It's your body, your choice.