I'm feeling super guilty at the moment for it too. It's like he's expecting me to leave him or dump him somewhere or with someone. When Mum visited for 30min today she was getting ready to leave and grabbed him and started saying bye bye and he just started screaming thinking either she was taking him or I was going. He wouldn't stop until I took him back and then he stopped instantly. Poor little munchkin!
Oh well, it will all be over soon!