I think you're being completely reasonable. While I haven't been the parent in these circumstances, I have been the child. In my life, it was my stepfather's 3 children and his ex-wife. She did everything she could to make life difficult and worked the kids to a point where they refused to come and see us anymore. The advantage your DH has is that he can contact his kids independently from their mother - my stepdad was unable to so she easily convinced them he didn't care. From what you've said, it sounds like the kids know how you both feel about them and they feel comfortable in confiding in you.

The sucky downside is that you end up putting your life as a family of three on hold until you're all together. We couldn't go out for dinner, or go anywhere unless we were all together. So there was a big frustration build up from my brother and I, mum too, because we knew that his kids were getting the best of both worlds, while we didn't really have a life. We went from looking forward to them coming fortnightly to dreading the visit.

Can one of you take your DD swimming and the other go to soccer and then swap over the next time? That way the kids don't miss out on their activity and they won't feel that favouritsm is being shown (not that it is but I'm sure the ex is saying it is).

It's really hard. Hang in there, it sounds like you're doing an awesome job and making the best of a hard situation.