Peanut will be fine. I was emotionally abused by my ex for my whole pregnancy and DD turned out fine. I however wasn't. So try not to stress. I'm sorry for the loss of your grandpa. Unfortunately your baby won't wait.
Take comfort in the fact his soul will be a part of your new born. (one soul exits and another one enters)