It's definitely an age thing. A lot of babies sleep goes backwards or they start getting fussy when feeding when they hit 4 months of age. My DS1 started doing exactly what you described when he was 4 months of age. From what I've read, the best thing you can do is relax. Your bub will be fine (as frustrating it is for you!) and will be back to feeding well soon