It depends if they are doing it to control you or exert some kind of power over you. They are indirectly doing that, hence the phrase passive aggression. There is someone in my family who will ALWAYS keep people waiting, regardless of the severity of the situation. (I'm talking about someone dying in hospital and he *had* to have a shower - that kind of thing.) It isn't just a thing about being disorganised or whatever - it's purely a power play. "you can't go anywhere without me, you are depending on me, and I'm going to make you wait". Drives me nuts.
I still maintain this is assuming intent. And yet isn't this what we are trying to stamp out? People assuming they know what other people are thinking? Mothers feeling judged by other mothers etc etc etc... it's all about assuming what people are thinking. If people jump to horrible conclusions about people's behaviour all the time, we're not going to get very far.

Without the glint in the eye or the body language I'm not going to go around assuming because I'm unhappy with something, that the other person was out to upset me. What a sad thought process to live with