I think it is perfectly fine. As an adult, I am rarely asked what my middle name is. I only use it when I am filling out legal or 'official' documentation. My children are all aged under 7, and I would say that most people I know do not even know, remember- or even care- what their middle names are. Even my DD Godmother asked me a few years after DD birth 'what's her middle name ?'

I wanted to use DS1 middle name as DS2 first name, but DH didn't like the idea. He was worried that DS2 would think that we couldn't think of a better name and had to re-use one. But, George is obviously a special name to you, so I think it isn't weird at all.
If you feel OK about it, I reckon you should go for it.