I have been through that too. I think we all must get scared!
I felt like that well and truly even after 12 weeks. I think that up until you have bubs in your arms - you will feel some sort of fear.