hun. SOunds like a difficult situation.
I personally would still be arranging access visits legally, this way should something happen (eg. drug use) you've got something to fall back on.

Other than that I don't have any real advice for you. Just loads of and support.
Remember you are a stong woman and can do this,you are a great mum and your kids will always love you. xx