Thomas (Tom) I love it! I'm considering it for the 2nd name for this baby (my second boy)
Oliver (Olly) I love this one too. Sounds great with Samuel too. Maybe Trevor doesn't sound that great with it, but not that bad either.
Byron - Once again, I love it. Was on my shortlist. Works with middle name better than first two and sounds great with Samuel too.
Maxwell (Max) - Max is too common for my liking (though Oliver is pretty common too). It's not a bad name, but I really prefer your other three.
Only other thing I can think of to suggest it Nathan or Nathaniel. Nate as nickname.