Go with the gut, yeah. We pulled DS out of daycare cause it just wasn't working for any of us. I wish we'd listened to our instincts from the get go - we would have saved ourself some grief. I don't think DS is forever traumatised by the experience, but it has had a lasting impact. I can see it.
Good luck making other arrangements!