I was the same with my first baby. Don't worry. I am sure you're not hurting him. The nurses in the hospital and CHN's just "throw" them around (not literally) but they're no where near as gentle as the mother's. I am sure you're being very gentle.

Just like jasp said, when they're a second and subsequent baby their mettle is really tested. My DS2 is constantly poked and prodded all day. Stuff that would have TOTALLY freaked me out if someone had done it to DS1. When he's not being poked and prodded, DS2 is looking for DS1 because the excitement has ended.

Babies are very resilient and I am sure you're not hurting the little fella at all.