I don't think you can hold back money for something that isn't working, it's not like renting and doesn't work that way. I would trust what your settlement agent is telling you, whilst he may have been rude, he's probably right.