I think I must make mine differently - you have elastic on the outside and inside?

I used to use a thicker elastic at the back, like you, but have found that the same thinner one as on the legs works better for me - something you could try.

I just sew the end down with a straight stitch. Then pull, sew a bit with the zigzag/elastic stitch, pull and sew a bit more and so on, then sew the end with a straight stitch.