Incest among other things is often about lies and indirectness. A 'secret' relationship that can't be discussed; a parent or other married adult who looks monogamous but isn't, etc. Also there are some relationships that can be stifling but not sexually inappropriate. Still, if you have heard a report that this woman is abusive from a family member who is in a position to know, the best thing is to discuss privately with your boyfriend. Whether there has been sexual abuse or not, it sounds like he needs to psychologically separate from his mother. If there has been sexual abuse it will be harder.
Confronting an abuser is almost never a good idea until the survivor is ready for it, assuming the accusation is true. The only exception is when there are children at risk. If there are no children at risk, in many cases it is best to avoid a confrontation and focus on 'living well is the best revenge.' That is b/c family rarely responds well to an accusation. People often get really upset or ignore it and that can be frightening for the survivor.
As with other things, the issue is with you and your boyfriend, not you and his mother, even if his mother treated him terribly. However if he remains close and you have children and you feel the accusation is true - from talking to him and other family members - you will have to be careful about your kids. It can be a hard life.
It comes down to what your boyfriend says, but if his brother got emancipated at a young age due to similar behavior, there may be documentation of abuse. Still the decision to confront is up to your boyfriend. Some people are falsely accused and others did abuse but genuinely don't remember it - maybe they were alcoholics or etc.
good luck. there is a book by Laura Davis called allies in healing that talks about these issues. Loved ones often want to confront an abuser but unfortunately that is rarely a good idea.