That's OK - the fencer can fix it then for his mistake.

And yes, 4" on a long boundary is a LOT of land, I'd be complaining too. Plus, they've done a lot of stuff without your permission and trespassed on your property, again, complain-worthy.