Hun, it is so normal to feel what you are feeling.
I don't think there are many people out there who wouldn't feel a bit peeved at a guy who walked away, and then comes back & gets to do all the fun bits for a few hours a week - but doesn't have to deal with the ins & outs of everyday parenting on your own - not to mention the past few months when he wasn't there at all.
It's not easy! and you are doing such a fantastic job.

And you're right, he probably doesn't get it at all, it's just one of those things. So don't feel bad at all for how you feel!

One thing I might suggest is that if he is pushing the envelope a bit in terms of being flexible (and I don't know, it might have been a one off?), you might want to get something in place saying when he is to have her. It's nice to keep it informal & flexible if it's working for you, but if it stops working, don't hesitate to be firm. We had to do this years ago because XP used to take advantage in a big way & in the end it was easier to say 'no- this is your weekend etc'. I would have liked to be easier about it (and we tried that) but it just wasn't going to work.
Like I said, it may have been a one off & if you can keep it casual that's much nicer for everyone - but if you have made plans etc don't feel at all like you have to change it to suit him.