Thanks to all of you who replied I appreciate it.

We have decided on Hong Kong! We think it just might be a bit better to take young kids to, even though it will be busy, we are going for the food, culture & shopping aswell as Disneyland for our 3yr old!!!! (she'll nearly be 4!). She is already talking about it.
Vietnam might wait for a few more years.....

Dirty cities and places don't bother us too much, we've travelled to different parts of Asia a few times and seen a few things so it doesn't phase us. We'll only be visiting for 5 nights on this leg of the trip anyway.