My DH was the same & I remember feeling very worried!! He never voluntarily touched my belly. He only felt DS kicking when I told him to & even then didn't leave his hand on me for long. He just didn't seem excited at all. I remember feeling very rejected too.
The only time he ever seemed excited was when we heard the heartbeat, that made it "real" to him.
It's not indicative of how they will behave once the baby is here. My DH is completely different now that DS has arrived.