you know what, its only a problem if YOU see it as one!
as you said he is 5 months old! if he was 15 years old, well then it would be understandable for them to have a problem with it.
he is your baby not theres, if its not a prob for you, well tell them to but out! after all he is a baby ffs!!!