Do you have set times for him to come and visit? I'm not sure how you work it, but if it was possible for him to visit with bubs - whilst you sleep. Even one hour, 3 times a week could make a difference for you, and you need to rest!

Yeah, course he is now the parenting EXPERT. It's only to undermine your confidence bit by bit. Can you imagine how you would feel as a parent if you were well fed and rested and could swing by and spend a bit of time with the kids when you felt like it, and hand them back when they smell? (This is coming from someone with three littlies and NO SLEEP so you might not feel the same!).

If he comes to visit, let it be your terms (sounds like they are anyway) and beneficial to you too. I good way to gauge if he is bonding with bubs or doing it to manipulate you is to not see him when he visits.
My exH visits to my daugher (not his though) dropped off noticeably after I made myself scarce (also to avoid his stupid,"harmless" comments) during the visit.

Good luck with everything, I hope you have another good nights sleep tonight...xoxoxoxo