I wonder if it is muscular? I haven't experienced it personally, but maybe the cold tightens up the muscles or something and makes them a bit achey? (Particularly because they're doing some pretty serious stretching atm anyway...)
Sorry that it spoiled your swim