I had exactly those cravings during my whole pregnancy. I would have done it too, if I could have been sure it was completely safe for DD. I was even to the point of wondering if you could sterilize dirt in the microwave! LOL The closest I could get was the chalky texture of antacid tablets - not at all satisfying. Don't worry, it's not uncommon. Just frustrating because you can't do anything about it.
As Sarah said, some say it is a deficiency - probably calcium. Lots of little kids try dirt and sand too.