No I don't think you're being ridiculous at all!

I chose not to have it done with either of my kids. Choosing NOT to have a test done is as valid a choice as having it - especially if the outcome would make no difference to your decision making.