Can you fit the crib in your room near your bed then he still has his bed but you and your DH can have some space back for a snuggle as well. He may feel more comfort still been in the same room as you.

hugs xoxo