Simone, I agree with your Dad. I'm sure that you can get payments from centrelink. Is your ex payng child support at the moment? Your priority in this should be yourself and your children. Your ex certainly doesn't make you a priority or teat you with any decency - why should you treat him with kid gloves.
On a practical level I don't think that you should be alone with him at any time. If you want to drop the kids off or discuss anything please do it with a third person person present preferably in a public place. I don't want to scare you but one of the most dangerous times for a woman in an abusive relationship is when she is trying to leave it. It might be best for now if you take steps to formalise your custody of the children and also for the interim request that he only be granted supervised access.
I hope that things get brighter for you soon. You are doing the right thing for yourself and your kids. Stay strong.