I couldn't stand the smell of tap water when I was pregnant, so we had large 15L bottles delivered to the house - cheaper than buying it at the supermarket.

Then we switched over to a Stefani ceramic filter - bought the bench-top set-up at Bunnings for about $50. Have been pretty pleased with the result. Perhaps the filter you were using wasn't a good one?