It's not like there is a handbook on how to handle this situation chicky babe.

Don't kick yourself too much,

Maybe you can see a Family Lawyer about what you can do about this - it's not fair that you are at the whim of his 'parental' feelings whilst trying to grow the little tyke...