I have heard and read many times that if you retract it back too early, or force it in any way then it can cause some damage... Most things I have been told or read seem to say around 4 or 5 is an appropriate age to start.

Though I know a generation or 2 ago it was entirely different, as I'm pretty sure MIL was doing it to her sons foreskins when they were just babies.