It bothers you so much because you don't want to see your DD get hurt by him, either now or in the future...

You're her mum and protecting her - totally natural. You want what is best for her. Which is not being stuffed around by her own father!