I'm a bit like Olive, mine goes down pretty much straight away. And I'm the opposite to OP, my uterus shrinks really quickly post birth.
Every body is different. It will get there.