Yep it's a build up of soap scum/fabric softener.. I didn't know until mine started doing it recently but we're supposed to use a descaler in our washing machines every so often just like our dishwashers need a clean out.. Bicarb and vinegar through a hot wash is great to maintain it but usually once it gets to the point of the brown flakes, the Bicarb vinegar washes only last so long.. You need to get a proper washing machine descaler like Scalex, we got ours from Bunnings, 2 rinses later and no more flakes! After that if you do a hot wash with Bicarb and vinegar once a month and use the descaler every 3 - 12 months depending on which descaler you use and the strength of it, you'll keep the flakes away apparently.. I font even think we'll need to do the descaler that often really as it took 5 years for our machine to start 'flaking' and yep I'm on my last bottle of fab soft now and then I'll just be using vinegar and essential oils.. It's apparently quite bad for our machines and thats what the build up is, all the fab soft clogging. I have noticed my clothes are softer than ever with out using softener since I put the descaler through anyway!

Hope that made sense and isn't too all over the place..

Good luck!
^ Yeah that !