I haven't had my first child yet, however, I used to work in childcare and there seemed to be alot of children in my room who had the as comforters for sleep time or when they were upset for one reason or another (age group - 1.5 yrs - 2.5yrs).
However, maybe if you took it away when the baby was 6 mnths or so it wouldn't be so difficult as long as you replace it with something else.
HTH

Good Luck with your decision.