Personally I'd leave it. DD has done similar before, where she'd remember an accident from a week ago and her finger/arm/leg/bum would be sore. In those cases I knew she wasn't still sore, but with your DS it sounds like he isn't and maybe you're right about it being sore when it's cold and immobile.

But, given it's not affecting him and the pain in the butt it is to go to the Dr and have them say "yep, I think it's s bruise, it'll get better soon" I'd wait and see if it got worse or took too long to go away.