To me it does sound like more of a nickname than anything else, all other associations aside. But not really a reason not to use it if you really like it - I know I've heard it used by grown men as well.