Its normal to be worried, but really - try not to feel guilty - because youre not doing anything wrong!
Even though you are DS's main carer, there is certainly nothing wrong with leaving him with his daddy for a day, in fact it will probably be great bonding time for them
I know its hard the first time you leave them - but try to relax and enjoy yourself
You'll probably cry more than he does, I know I did the first time I left DD#1!