With my first labour, I had an epi, so I too was a little worried/curious about what pushing actually felt like.
With my second labour, my waters were broken at 3am when I was 5cm dilated, and about 40 min later I felt that overwhelming urge to push. It was like I needed to do a massive poo! The midwife did an internal, I was fully dilated and my cervix was a little over bub's head, so she moved that out of the way, and I started pushing. She said to me to push with my bum, as if I was doing a poo. I could feel bub moving down, and she had been posterior right up until then, then she turned, and I felt that too. Very ouchie though!
Pushing is the most amazing experience I have ever encounted. I hope you're able to go through your next labour without the epi so you can experience that.