Personally I think that even though you can't be expected to drop everything and make your life fit in around your baby, I think your baby is just as entitled to have the world revolve around them for at least one day of the week where they can just be at home, relaxed even if they don't generally sleep for long, and not be dragged from pillar to post. I never expected my 4th baby to just 'fit in' and get used to sleeping on the go all the time. They have just as much need (a right even) to have a day without going anywhere just as much as the older kids need all their after school activities and you need a coffee date kwim? Once they are predictable with their sleeping to an extent, I always planned my day around that. I know it's a lot easier for me out here where there is no pressure to have kids doing activities 5 days a week and I'm not stuck in a car for 40mins a day taking kids to school, I could do it to an extent, but I don't because I know how ratty and overtired bubs always gets from being out all the time so I avoided it because it makes life easier for everyone that way. if it were me I would be looking at making a few changes to be a little more accomodating for him - like with coffee dates, make some of them at night time after dinner once he's in bed and DH can look after him or even have them at home where one person picks up the coffees and cake on the way to your place and visa versa.

If you do have so much on socially it will be near impossible to get him into a routine anyway as it will nearly always be broken because you have to be somewhere, unless you make some changes to fit everybody in. Maybe change the girls' activities to afternoon ones so you have the morning at home to give him time to sleep etc.