For the night feed, is he lying down? That might be why it's ok then.
Or, is there any difference in volume or speed of flow on the right? I know for a while DS refused the left as there was less milk and a slower flow. Again, he might not mind so much at night so that's why it's ok then.

I'd get him checked out to eliminate any physical problems on his end.

I think so long as you don't get blockages, your only problem would be lopsidedness...

ETA - yes, great typing