I think ignoring & distraction is the best answer at that age. If you do anything to acknowledge it, he's getting a response & thinks its amusing. I'd just move his hand & not say anything iykwim. I don't think you need to move away, keep doing what you were doing, just stop whatever he is doing without letting him know thats what your doing. Then find a book or toy to distract.
Does that make sense?