I have never really had a relationship with my father, and yes it hurt like mad. Even now I'm not completely okay with. Whether I'm a well-adjusted person or not . . . i dont know, lol! But I did read somewhere that a childs first relationship of trust is ALWAYS with the primary caregiver, and their self-esteem is based on their relationship with the opposite sex parent. So if that's true, it means that as long as YOU are there for your son, he should still have strong self-esteem and be confident.