If they call again you can say that you're not prepared to offer a reference. Refusing to be a referee says volumes without saying a thing.

My Dad rang someone for a reference once and he said 'I tried him. Now you can'. Lol. We didn't try him.