In situations like this I figure I'd rather report someone and risk there not being any problems, than not report and have something horrible to happen to a child.

As Keike mentioned, they don't just take children away for no reason and they certainly don't keep them separated from their mother/father without a very good reason. And I agree that if the first child was adopted out, things must be REALLY bad so definitely report.