I usually just chuck them in with the rest. They seem fine to me?
The white fleece does go ikky grey, but I have some green fleece liners & you can't notice it on them at all