Really, dont worry to much about it. I had the same thing with my XP. He was determined to get 50/50 custody of our daughter (we were at court for the first time when she was 6 weeks old). The judge all but laughed at him.
But definatly record EVERYTHING that happens, who has him, what times they have him, anything unusual that you notice or are concerned about, phone calls XP makes, anything that XP might contribute towards your son. Especially make sure you keep track of everytime he cancels his visitation.

My DD1 is 5 and i have 5 diaries (1 for each year) so far that have recorded everything from day 1 of our negotiagtions about her care. Its come in very useful over the years when XP has wanted to be an idiot and tried to pull things over me.

It might be in your interests though to arrange activities in your area, playgroups, swimming lessons, play dates, so that IF it does come to that, then you can just say that you have established friends for your son and he is used to doing these things with you at these times.