When I was young and single with a baby (allll those years ago lol), I got sick of people assuming the worst so I just lied my butt off! What does it matter to half these idiots that will do nothing for you and your bubs but enjoy judging others for something to do or gossip about?
You can say he's overseas 'on a big contract'. Peacekeeping in *insert location, or if they are the really nasty type tell them babys father was killed in a tragic accident (on the eve of your wedding for even more fun!) - cue tears.

Otherwise you can tell them it turned out he was a right w@nker and you couldn't imagine going any further with him like some people you know who stay in a crap relationship 'cos they are too scared to do it on their own...