Do you chart or use OPKs or check your cervix? Maybe one of these can help you pinpoint ovulation. Theres a strong chance you are ovulating later than you think.

Not too sure about the brown CM as I've never had that, though I think some people have brown CM around the time they are ovulating (and if its more like spotting it can also be a sign of implantation).