I did it about 15 years ago. We didn't have kids yet so it was a bit different but still hard for our families. My ILs are up here too now, and my family are scattered and used to it, but it was hard at first.

I think you need to bite the bullet and just tell them. Hopefully they will be happy for you and maybe they'll use the holiday house more knowing you're nearby. Sometimes you just need to do things for yourselves and I'm sure your family will understand that.