My opinion is that it is ok. If you are just going from the car to the house and back, he is going to be rugged up and will be in your arms next to your body. Sitting outside in the wind is a different matter but I would be fine with what you have said. Isn't it difficult when you get people with strong different opinions. It makes it hard to form your own.