Has this manager not heard of the Sex Discrimination Act? It is illegal to harrass or discriminate against someone on the grounds of pregnancy.

He has no right to intimidate you about your appointments. You have been honest and upfront with him and your work is up to date. He is the one who is in the wrong.

Just make sure you keep a record for yourself (just notes in a diary would be fine).