Swansea sounds like a really good idea! For just three nights you can just focus on visiting Freycinet that way. Lovely!

Otherwise Binalong Bay would be my other suggestion for a romantic weekend. The Angasi restaurant is great. It's a longer drive though. Anywhere on the east coast is stunning.