It is the BPA in the bottles that can cause harm. The chance is very minute (don't know the exact figure), but the risk is there.

You can get BPA free bottles or go with glass if this does concern you, but there are many many more people who use plastic ones. Like the whole picture of parenting, you need to do what's right for you..