I didn't wanna read (again) and not respond, so here is my 2 cents

You mentioned sitting your ex down and talking to him - has he proven himself capable of proper adult conversation before? There's no need to put yourself through it again darl; discuss it with your son, and make the change. Would EXH even notice? Does he actually go to the games?

I don't know if you're already doing it, but keep a journal of the things he says to the kids that you find manipulative - I think they would go much further in a custody hearing than the location of the boys sporting activities. Bug hugs to you