First and foremost - try to rejig your attitude towards yourself for a bit. It almost feels like you are apologetic about your situation. It is what it is - that's all. Better than living with the silly twat and waiting for him to come home drunk every night (random example on my behalf ).
It's hard not to take it personally that he appears not to care too much for you and seeing Grace, but that's not about you, it's just that he is a twat. (oh how I love that word).

Second - remember no matter what, you can't avoid him being a....you guessed it - twat. Catering to his needs in order to avoid a negative backlash just makes it worse. Every change has a negative repercussion.
ie - I want to change the fact my floors are gross. So I have to mop them (afaik, thats negative), like it or not. So I get through the mopping, now my floors are clean.
I get what I want but I have to do the crap bits.

Thirdly, look for Family Relationship advice. Maybe google it, or even call C'link as they should have a list of places to help you out. Go see them to find out what's fair and reasonable, then present it to his lordship and if he doesn't like it he can take steps of his own to get what he feels is right (go to mediation etc).
Once you feel confident about where you stand it will be easier to implement and stick to.

And tell his friends to F off, it's an access visit, not a freakin zoo...