Have you looked inside the wardrobe to see if any clothes (especially those not worn recently) are covered with mould? A few years ago now we lived in a poorly ventilated flat and had mould growing on the ceiling, clothes etc. We bought one of those closet camel things to put in the wardrobe and it is amazing how much moisture it absorbed.

Does the house have cross ventilation?