I think it would be fine.
I know the reason you're not supposed to have hot showers/baths is because it can raise your body temperature, which can be dangerous for bubs, but i've never heard of it being bad to swim in cold water.