Will you be here on a Sunday? $2.50 tickets (under 4s are free) on trains/buses/ferries/light rail. You can catch train to central, get on light rail to The Star, visit Zumbo for yumminess (plus other great eats in the food court there or the Garden Buffet, which is pretty amazing) and nice clean toilets! Then walk to Darling Harbour and explore, or catch the ferry from near the maritime museum. One of the routes from there goes via Luna Park wharf, under bridge to circular quay.

I would stay near Darling Harbour (including Astral Towers at The Star - very nice) or near the Quay. Are you on a budget or splurging?

Are you into food/shopping/sightseeing the famous bits/museums and culture/architecture?

I think you have an almost 4 yo and a 1 yo from your sig?

Chinese Gardens are great
Sydney Tower (eat a meal at yum cha in Westfield Sydney or in the food court on the level below)
Darling quarter has a fantastic playground (take spare clothes, towel and plastic bag - it has a water section)
Darling harbour nice but with the redevelopment isn't quite as pretty as normal
Botanic Gardens are beautiful
Walk around near Opera House
The Rocks (not as interesting for kids though)
Shopping in the QVB (go to the upper level for the clock etc, which is pretty cool), Pitt St mall, Westfield (under Sydney Tower) and The Strand Arcade
You might want to look at some of the multi-attraction passes if you want to see a lot
Museums tend to be cheaper - Powerhouse is pretty cool for your 4yo and there is also the Australian Museum

Hope that gives you some ideas! Any more questions just ask.