Plastic tubs are a good idea. Even things like paperwork and teatowels need to be locked away. I found mice nesting in my bills drawer and some had eaten holes in a few tea towels.

We have cats. When we first moved in, we would have mice climbing the curtains and running across the kitchen, now we only see them in the house if a cat brings them in. Chances are there might be a some feral cats hanging around, so you may not have to get one yourself.

They dislike peppermint, and I think oil of cloves, so wiping down the inside of cupboards, drawers etc is a good way of deterring them as well.