OK, sounds like they are going out of their way to be nasty. I take my previous sensible comment back... build the fence. I am sure you yard will look better with 6ft fencing all the way around anyway.