I don't think it matters so much when you have the crystal right there on your person, as its energy is inside your own energy field and able to interact with it. Also, bearing in mind that your aura extends some way out from your body, having a crystal close by but not touching you (eg on the bedside table, on your desk) can still have a positive effect.

Where size does make a difference is where you have a crystal that you are using in a room to effect the energy of the room or of a larger group of people in it. Then it's better to have a crystal that has a larger energy field, like a large chunk or cluster or cave.