I think because you have the option of just sending DH and the older kids on the trip then it's OK. It's not like you're making the entire family miss out and DS2 is still so young that he wouldn't even remember the trip so it's not like he's missing out. I think under the circumstances it's the right thing to do.