I can second Wye River my family has a house and we go there all the time. The beach has life savers on weekends and school holidays, it's in an inlet so more protected and the pub is kid friendly. It's only 2.5 hours from Melbourne and you can go on day trips to explore the area like the otway fly and we did the 12 apostles last time.