Found this that explains the quirks of section 27's in Vic a bit better.

Once you sign the Contract, you have an 'equitable priority' established in the property and can place a Caveat on the Title.

Many Solicitors and Conveyancers suggest doing this as this prevents any further dealing with the property by the vendor between signing the contract and settlement, without your express permission.

You cannot unreasonably withhold permission for the Agent to release the Deposit once Requistions on Title are answered to your satisfaction.

You satisfaction should include that there will be sufficient funds at settlement to clear the balance of the mortgage (if any) which may be registered over the title. You should also be satisfied that the vendor is not in arrears of the mortgage loan.

There may be other issues which may cause you to withhold permission, but if you have paid a deposit - and particularly, if you have signed the S.27 then the Agent will prepare an Account Sale to the vendor and the funds will be released from their Trust Account unless you (your representative) advises in writing that permission has not been granted.

No other State in Australia has an early release of deposit clause, and is one reason why Victorians are using Deposit Bonds or negotiating smaller deposits.

It is often difficult for a purchase to say 'no' to paying over a 10% deposit when they have it, mainly because they feel 'mean' if they don't pay the full asking deposit. Purchasers with smaller deposits are often in a stronger emotional position to argue for a smaller deposit to be paid or to lodge a Bond

However, if you have paid the deposit then unless there is a real and material reason you cannot refuse the release of the deposit. After all, the vendor may require the money to pay a deposit on their next purchase and this may be why the Agent asked you to pay a deposit.