Oh I wouldnt be keeping it down.. You have every right to be annoyed.

I don't know what else to say really. Maybe sit him down and say either help or I will hire someone to help (even if you don't mean it)