I can relate to a lot of things in your post Bath, especially after a near miss lately, only thing is that I've had my licence for almost 10 years.

What is the most worrying aspect of driving? That I might have an accident and hurt my DD, DP or myself
Did you have any dramas getting your license? Nope, my dad gave me lessons and I got my licence in a country town where the police officer takes you for the test because there's no transport dept.
Have you had an accident? No, I had a near miss a couple of months ago with DD in the car - I was going straight through a roundabout in the left lane on the edge of an industrial area when a huge road train coming from the left didn't stop at the roundabout. I had to swerve into the other lane, lucky there wasn't a car in it otherwise he would've sandwiched us. I still have anxiety about getting into the car sometimes but I am glad that I actually kept driving that day instead of calling DP to come and get us. It has made me so much more cautious and wary but I see that as a good thing - You have to be ready to react at all times.
If you accidentally killed a pedestrian (say a child suddenly stepped out infront of you) would you give up driving? (<--- that is my worst fear) I don't know, I would like to think not, but I imagine that it is something that would eat away at you.

Ultimately if you want to get your licence you need to do it for your own reasons and not because everyone else thinks it is a good idea. Good luck!