I never assume that I will be bringing any food home from take a plate things unless the host insists. Mind you, I usually dump anything left over in the bin anyway - most places that are bring a plate mean food has been sitting out for ages and lots of people have touched it or hovered over it, other things have been dropped on it or it just generally looks unappetising by the end of the event.
Like most others sometimes taking home bits and pieces of food is more of an inconvenience. As long as I get my platter/bowl back I am happy.

What I do get annoyed at like someone else has mentioned is when my food is not served. This happened recently to me over Easter when I made the most delicious chocolate cake and it wasnt bought out. I was so annoyed.