We had lots with DS, this time we're trying to limit it. I think ultrasound is a very important diagnostic tool when used judiciously during pregnancy.
In my opinion, the 20-week ultrasound is a good idea for all pregnancies, but that's probably the only one that is really needed, barring other medical indications.
But there are lots of reasons people have routine ultrasounds - it can be very reassuring, especially in cases where parents are particularly anxious and not yet feeling regular movements. I don't think the evidence is conclusive either way, so it's probably not fair to judge parents that decide to get lots of scans, particularly when many private-practice OBs will offer them as routine (which obviously suggests they see no risk in having many scans).

We've had 3 this pregnancy. 1 early on due to bleeding. Then we had our official 7-week one to confirm viability (standard after IVF). Then the 20-week one. We won't have more unless complications arise.

I would be more concerned about your OB bullying you than whether your not the 20-week ultrasound is worthwhile doing. All the best