As you mentioned I think you know what is best to do for yourself and your children,
You can get court orders etc, where he can be with the kids under supervision from a mutual friend etc, my G/F does that now.

There is lots of assitance if you look into it at centrelink.

Best of Luck.. you will make the right decision and go with it.