I don't think much of men who pull this on the mothers of their children. I mean, excuse me, but inside your pants is a pen!s, and as it belongs to you, you need to take some responsibility here... no matter who is going to have custody. Also, if he had taken the kids, does he honestly think anyone would believe he'd be able to do it without the help of his mother and sister?? NO! He'd STILL be out playing poker and leaving the kids behind, so don't allow him to pull that on you. That's just a throw away line that no-one buys. It's p!ss weak, actually. He can use his childhood as an excuse till the cows come home, but it's pathetic if he's going to use that as an excuse to be a loafer for the rest of his life.
It's best for now you don't spend TOO much amicable time with XH, Sim. It's great that things are relatively calm in proceedings. I know you two have the potential to be friends after all this is done (I wouldn't say it if I didn't mean it), but I think the focus should be on working out the settlement, keeping it civil and THEN sort out the friendship parameters.
Now, you need to figure out why some other fella out there should think you're worth hanging out with. You need to be comfortable in your own company and to appreciate why you are good company before you can offer it to someone else and hope for good outcomes It's not easy street, but it will be worth it