I don't see a problem with it. Because you've already used it twice it's clearly a special name to you and I think it actually makes it special for the child who scores it as a first name.

I can imagine if anyone does ask the older boys their middle name it could go something like:
George
Oh, your brother's name?
Yep, it's a family name

No biggie