I think that if you don't talk to him there is no oportunity for change, for him to ever acknowledge what he probably already knows on some level.
Even if you just said "the fact that you weren't ever there when I needed you most means that I will never be able to trust you or be comfortable with you now, we can't be friends until I know that you understand what happened and where I've come from", at least you're saying something, and you may find your way to some kind of closure.
If not you won't be any worse off than you are now.