Have you tried talking to him about it, directly? I experienced the same thing really early on and it was as much of a shock to me as it was to him. We didn't talk about it at all for the first few months, but the underlying tension was so present that we had to sit down and discuss how we were both feeling, honestly.

It was difficult to articulate how I had become this person who was quite indifferent to sex and intimate contact, but he understood. And he explained to me that he felt as though he was being rejected as a result of my indifference, and that in fact despite how I was feeling (i.e. slightly awkward and constantly nauseous in those early months) he loved me and was attracted to me. Perhaps if you think about his behaviour as testament to his desire for you, it makes a bit more sense?

I came to realise that men can't really ever understand how physically taxing pregnancy and new motherhood can be, at any stage, so his initial sulkiness made a little more sense to me then. But...it was difficult for a while there, and he was hurt and upset and cranky, at times. I really think speaking about it openly, and clearing the air, might help things. Personally, I don't think you should have to feel as though you need to 'give in' to his advances - imagine how you would feel if the situation were reversed, and you discovered that he had been, essentially, suffering through it just because he knew that you wanted to? That would feel quite crushing.

Really, I think the best thing is to talk about it - what do you have to lose? And imagine how much better you will feel when you no longer feel weighed down by guilt, and he by confusion? Neither emotion is a known aphrodisiac, anyway! Good luck!