Tell them your doctor says no work stress or contact
I had minimal contact from my work while on leave, just a phone call or two to keep me up to date with people leaving etc.
I'd suggest you ring your boss, or supervisor, and let them know while you'd love to help your doctor has told you that you're not to be exposed to work stress. Nothing major is wrong, you just need rest and relaxation. They can then pass that on to the people who keep calling. Your boss/supervisor may not even be aware they are contacting you.
Good luck with your lovely new bubby.