Yep totally had this problem with DS who is not the greatest of sleepers anyway...when DH was home he was hopeless.
Doesn't help that DH stomps around the house like a freaking elephant forgetting there are babies trying to sleep