Are you wearing him on the back?
DS was happy to sit in it, poke his head out and have a look when he was ready and loved that he could 'turn the world off' at will in the front position. When he was ready to see more, I'd put him on my back and he was happy as larry that way. Easier on my back, too. I haven't used it for a while because DS now likes to walk a lot more, but when I was using it last we were overseas and there were times he would arch his back for it...so we let him walk till he was ready to hop back in So he can say he's walked the streets of London all by himself!
I won't be selling our Ergo, I know I'll be using it for the next child and patching up the little tear in the fabric (caused by a struggle!).