I do not think age matters at all!
Boys mature slowere than girls remember, and if he made that comment 2 years ago, I am sure he has grown up A LOT since then, and probably regrets the comment himself. I met a guy who was 28 when I was 17, and although nothing happened straight away, that was 7 years ago, and that man is now my husband and we have a DS and another on the way. I think age has no relevance at all and it depends on the individual person. As long as this boy treats your DD right! I especially dont think 4 years is a big deal at all!