lol, I think he's being a bit sneaky prob because he thinks you'd say no.
Yes, I would be annoyed. We always check with each other so that we know who's spending what and how much we have etc.