If they are smoking outside at your place, It is possible that they are bringing in smoke particles on their hands and clothes that your kids are reacting to.

I am always surprised and saddened when people won't miss out on a ciggarette, while in the presence of kids. Or who will go outside and have one, but then not wash their hands before holding or touching a baby.