I do the same, when I boil the water for the bottles I just pop some into a bowl & chuck the dummies in.
However if you do want one you can microwave the Avent Dummies can be put in the microwave, we just had a clear one first up....not sure about the one's with pics on them.