In the case of deli meats, listeria can be a concern when the meat has been sitting out in a temperature that will allow the bacteria to develop.

I think that a common sense approach is best.

If you ate deli meat that had been sitting out in the wrong temperature, then that is when there is a risk, and yes as people have said the risk is low, but it is there.
I think there are probably a number of items found in most people's fridges that could have actually have a higher risk of listeria - such as leftovers etc!