I would suggest either putting bub in bed with you, or maybe moving the co-sleeper a bit further away. I have one too, and you can put up that fourth side and have it like a mini cot. Its hard to move them further away, but it could be a good compromise for you.