I am pretty sure if they haven't done their job correctly they cannot make you pay it all back suddenly. I would get in touch with the relevant ombudsman and see what they have to say.