I guess it depends on how rough or gentle you have been as well.

Are you keeping a diary of dates? Maybe just jot down the times it has happened to see if it was just a once off thing or something that happens more often.