The fence goes on the fenceline. If he wants it moved, he can put it on his side but not on yours if you don't agree. And a retaining wall on the fenceline is actually the downhill side's responsibility. At least in our council area it is as we have had this problem being on the low side and the wall/fence was starting to lean towards our house. We had to fix the retaining wall ourselves but the fence was joint.
This. He cannt just move the fence. If he has an issue he would have to move the sleepers.