He sounds dodgy to me - trying to bully you to get what he wants. You have been very reasonable and he has given you no proof in return. Heres hoping that he will back down by himself.