Most men didnt get born with the 'tactful' part of their brain working unfortunatly. I am sure he didnt mean it the way it came out, and i can really understand how its difficult for you to move on from that comment.
From personal experience, i think you need to mention it to him, in order to help move on. Otherwise, you will be thinking lots about it, and if he really meant it or not, and this way, it will clear the air, and give him the opportunity to say sorry, and that he really didnt mean it the way that it sounded.
Hugs and sorry for your loss. Its no ones fault. Life just sux sometimes.