We didn't use sleeping bags, so I guess my answer is any time! We did layers of clothes as well. Knitted vests were great, pj's where the top buttons to the pants so it doesn't ride up, or one piece pj's with feet on them.

My kids just didn't take to sleeping bags at all.. they are like me.. like to stick their legs out the blankets hehe. No way I could handle my feet being trapped in a sleeping bag. Even now when camping, I open up my sleeping bag and use it as a blanket so I can poke a foot out LOL. So yeah.. warm clothes were the go here