This sounds identical to DH's brother, and I blame the wife for all of it. I think she has jealousy issues.
It's so hurtful, and you both have kids, they're cousins and yet never get to spend time together, it's so frustrating isn't it. Family is so important and especially as kids, having lots of cousins is wonderful. it's such a waste and a shame that your brother and his wife can't get past whatever issues they seem to have with you and just suck it up for the sake of the kids.