Perhaps you're not quite ready CK? It does all get easier as they move into the second year and he's probably at an age now when he can go without you if need be, so you might be able to work some compromises as you move toward weaning, if you want.
And when did he turn one!?