I'm sorry you have to go through this.
While you can't judge his interior state or his culpability, he's objectively going against the tenets of the faith in which he raised you. There may be factors that reduce culpability, there is no moral reason that compels a man with a wife and family to have a sexual relationship with another woman. That much, you can judge.
Your duty to honor your father does not prevent you from objecting to his plans. If you feel it is your duty to your mother and your siblings to bluntly object to what he's doing, then by all means, object! You can let him know that while it is no business of yours to judge him before God, he is objectively choosing his own self-indulgence over you, and that it is not acceptable to you to stand by and say nothing while he forces you and the rest of his children to watch their mother being humiliated in this way by their own father. I say "if you feel it is your duty", because this is a matter for some discernment, based on many factors you know but couldn't have time to go into here. You may want to pray about this, and consider these lines of Scripture:
"*If your brother sins (against you), go and tell him his fault between you and him alone. If he listens to you, you have won over your brother. If he does not listen, take one or two others along with you, so that 'every fact may be established on the testimony of two or three witnesses.' If he refuses to listen to them, tell the church. If he refuses to listen even to the church, then treat him as you would a Gentile or a tax collector." *Matt. 18:15-17
*"It is widely reported that there is immorality among you, and immorality of a kind not found even among pagans--a man living with his father's wife. And you are inflated with pride. Should you not rather have been sorrowful? The one who did this deed should be expelled from your midst. I, for my part, although absent in body but present in spirit, have already, as if present, pronounced judgment on the one who has committed this deed, in the name of (our) Lord Jesus: when you have gathered together and I am with you in spirit with the power of the Lord Jesus, you are to deliver this man to Satan for the destruction of his flesh, so that his spirit may be saved on the day of the Lord.
"Your boasting is not appropriate. Do you not know that a little yeast leavens all the dough? Clear out the old yeast, so that you may become a fresh batch of dough, inasmuch as you are unleavened. For our paschal lamb, Christ, has been sacrificed. Therefore let us celebrate the feast, not with the old yeast, the yeast of malice and wickedness, but with the unleavened bread of sincerity and truth.
"I wrote you in my letter not to associate with immoral people, not at all referring to the immoral of this world or the greedy and robbers or idolaters; for you would then have to leave the world. But I now write to you not to associate with anyone named a brother, if he is immoral, greedy, an idolater, a slanderer, a drunkard, or a robber, not even to eat with such a person. For why should I be judging outsiders? Is it not your business to judge those within? God will judge those outside. 'Purge the evil person from your midst.' *'
1 Cor. 5:1-13
He may know very well that what he's doing is wrong, but he also may be deceiving himself that his actions harm no one, that he "deserves to be happy", because he wants to follow the glamour of evil. Unless you have a compelling reason to remain silent, don't let that self-deception go unchallenged. If you have relatives that will help you, enlist their aid.
I would advise that you be ready to concede that he and your mother might possibly be in an invalid marriage, and that he might be free to marry in all good conscience at a later date. If a man finds himself in a marriage in which he cannot live in peace with his wife, there are moral ways to seek a separation and to inquire with the Church about whether his marriage was even valid.
Objectively speaking, though, he has not chosen that way. Let him know that you are disappointed in his behavior, in his willingness to hurt your mother and put his own soul and that of his mistress in danger, and that no amount of "you don't understand" is going to change that. You both understand his moral position very well. He'll do what he wants, but stick to your guns on that point.
If you refuse to associate with him while he is engaging in this kind of self-deception, there is nothing immoral or disrespectful about that. It is the consequence that both Our Lord and St. Paul recommended, in such a case. Unless your mother asks you to do otherwise for her sake, there is nothing in moral law that prevents that course of action, and even things in moral law that advocate in its favor.
*"Everyone who acknowledges me before others I will acknowledge before my heavenly Father. But whoever denies me before others, I will deny before my heavenly Father.
"Do not think that I have come to bring peace upon the earth. I have come to bring not peace but the sword. For I have come to set a man 'against his father, a daughter against her mother, and a daughter-in-law against her mother-in-law; and one's enemies will be those of his household.' "Whoever loves father or mother more than me is not worthy of me, and whoever loves son or daughter more than me is not worthy of me; and whoever does not take up his cross and follow after me is not worthy of me."* Matt 10: 32-38
If you decide not to raise this issue, do so because you discern it would be to the detriment of others in your family. As for you, always tell the truth in love, with patience, kindness and a hope for repentance if you admonish the sinner, but be willing to suffer in order to tell the truth, if it comes to that.
Good luck, and may the Holy Spirit guide your every word as you go through this.