Definately for me anyway, it was the conflicting advice. Whilst I was in hospital it seemed like every shift change would bring a new way of doing it 'properly' and a throw out of the method I had just been trying.
Also, a lot of the midwives I met through my struggles with breastfeeding were rude, uncaring, intimidating and abrupt and in the end I thought 'stuff it, I'm not battling with them any more...I just can't do this' and gave up.
I'm all for breastfeeding and I can understand where they're coming from (the midwives), but they need to be reminded that you catch more flies with honey than vinegar.

Edited to add...No one, not even my friends, told me that it was going to be excrutiangly painful to begin with!
This, to me, was the world's best kept secret!