Go with your instincts. If you feel that your DD will be in an unsafe environment (which from the sounds of things she would be) you have every right to say no.

Remember though, that DD's grandparents are her family just as much as her father is - if they are prepared to work within some agreed groundrules that you're comfortable with (like no smoking indoors when DD is around) then helping them to maintain a relationship with your daughter can have massive benefits. As difficult as it can be at times. It may also mean that they stop pushing so hard for your x to make demands etc.