That sounds very, very tricky to organise! I'm not sure. Do you have a backup person here to be with you in labour?
Why only 5 days? And why did your OB say he needed to be back earlier? I mean, is he expecting an earlier labour for some reason?
If not, it might make sense to push it out, to get the most time out of his leave post-baby. He might not make it for the birth - which would suck, but that's the risk, I guess. If you ended up going earlier still, then he might still miss it, you know?

Good luck!