Sounds a bit like is is competing for your attention and affection Mel? When DH is there he doesn't want to 'share' you iykwim.
Does he get any one on one time with just you or just your DH? Maybe it will help.
I am by no mean an expert, it just popped into my head when reading what you wrote.