Sounds like maybe he has responded to the drop in supply by feeding less, resulting in a further decrease in supply. It's quite possible he'll happily take the breast at that time even if there's nothing there - do you feel you have much milk then? Do you hear him swallowing? If he's getting formula and not demanding any other feeds then there probably isn't a great deal there, but it's hard to say really.
That's great to be getting some sleep! Totally understandable that you've had enough of the pump, too. That must be hard work.