You need to be careful when providing a reference for someone. I dont think that you are able to outright say anything negative about someone - but having said that I could be very wrong.

Maybe someone on HR could pop in and help you out
What is the point of being asked for a reference then? I am always honest if I am contacted about a former staff member. If they were always late and lazy, I am not going to say they were early and great at their job iykwim.

Definition of defamation in Australia:
You can defame someone if you say something false about them which spoils their good reputation, which makes people want to avoid them or which hurts them in their work or their profession.
I think as long as you're not saying anything false, then you aren't defaming anyone.