Anyway I suppose this is more a vent, but f anyone has similar experiences, yor advice would be greatly appreciated.
Yes, take it further as it IS a form of harrassment. I could be wrong, but I'd suggest he's not unaware that his comments are having an impact. Go back to your manager and ask her to take it up with him. It can be handled so there's no ill feeling, I'm sure. If he persists then it could be taken up in a more formal manner via an official complaint. Your manager has a duty of care to you and your colleagues.

Years ago someone I managed continually commented on my appearance in front of the rest of team that I managed - she made sure she was heard. Always it was "you look really tired, are you stressed", "you look like you had a late night, are you ok", "the hem of your skirt is showing (er, it was the design of the skirt)" - on and on it went. In isolation you might think she was concerned I was burning the candle at both ends, or got dressed in the dark (!). But there was a pattern to it and I realised fairly quickly it was to undermine me with the rest of my team. I ended up calling her on it and asked her specifically not to comment on my appearance in future. She ended up losing her job, not for that but for many other reasons. But still, whether it's a manager or someone you manage, it's not on.