If you (or friend) don't want him to go, say so. If you don't he will assume there are no issues.
That's the key really, isn't it? You need to be clear with each other and your friends what your expectations and needs are, be honest and up front and work out how to get what you need, and how he can get what he needs also. There are no issues, no matter what you are or aren't comfortable with, if there is honest communication and understanding happening.