I told my boss and my boss' boss, and DH told his boss. And guess what? Two of the three of them have done IVF. You may be surprised at the sensitivity you may receive, and personally I would rather my boss knew so that would be one less thing I'd have to stress about. IVF is stressful enough without having to worry about whether work thinks you're bludging. :P