Oh . I haven't done it yet myself (eeep!) but I agree with the others - he'll have a ball with daddy, it's you who'll suffer, if you let yourself!

Even if you're the main carer, he obviously has a good bond with his daddy too and feels safe and secure with him. That's why I'm sure he'll be fine and definately won't be scarred by the experience.