My dad was never in my life, my mum raised me on her own. Look at me now, im doing extremley well in life with a son of my own.
My mum met her ex-husband when i was 5 and at 8 they got married, i didnt have much to do with him, and barley knew him. My sister was born when i was 9 turning 10, as she was born with down syndrome my parents had a hard time coping, (step-dad ended up having an affair), i vowed never to let him get close to me after that.
To cut a long story short, they got back together had the twins (boy and girl) and i didnt get close to him until i was 17 when i needed a father figure in my life.
My step dad was never a real played a good father roll, and my parents seperated a week after my 19th b/day.

I dont regret not having a "father" figure in my life (although had male living with us [step-dad]), i dont think all males are like that, some just arent ready to be fathers when needed.
I dont hate my real father or my step dad, am just dissapointed in them. but never am angry or regret the decisions my mum made.
My mum never said a bad thing about my real dad either. Your son will grow up knowing he has a dad, but as long as he has you and everything he needs, it shouldnt bother him. and i know when the time comes that he wants his dad in his life, he knows you'll be by his side every step of they way supporting him, which he will cherish.
There will be hard times for him, but they would be hard times for him even with his dad involved.
You sound like you are doing a great job with him! Having his grandparents involved aswell is fantastic (i never had that). Dont stress, your son will grow up to be a fine young man, with just you by his side!

Hope this helps some what.....Take Care