I felt the same way with Maddy at first, and even though she is now at daycare full-time and for relatively long days; you could always do a shorter Friday with him to start...perhaps to ease you both in? Perhaps drop him off at 8:30 or so and pick him up around 3pm? That was what our dc recommended when Maddy started so it wasn't too overwhelming for her.

Good luck hon!