You seem to understand that adultery is a grave sin. You should also be aware that “Divorce is a grave offense against the natural law. It claims to break the contract, to which the spouses freely consented, to live with each other till death.” (CCC 2384).
Have you sought marriage counseling? My recommendation is to seek forgiveness through the Sacrament of Penance, and ***strive to repair your marriage. *** Try to convince your husband to attend counseling with you. God forgives us as we forgive others.
From a Catholic perpective, “a marriage which is ratified and consummated cannot be dissolved by any human power or by any cause other than death.” (1983 CIC, c. 1141).
Furthermore, there must be lawful reason to severe conjugal life with your spouse. According the Church, “spouses have the obligation and the right to maintain their common conjugal life, unless a lawful reason excuses them.” (1983 CIC, c. 1151).
Lawful reasons for separation (note: separation is not divorce) include: adultery by your spouse (see below), spouse who occasions grave danger of soul or body to the other or to the children, or a spouse who otherwise makes the common life unduly difficult.
In cases of adultery, "it is earnestly recommended that a spouse, motivated by christian charity and solicitous for the good of the family, should not refuse to pardon an adulterous partner and*** should not sunder the conjugal life.*** Nevertheless, if that spouse has not either expressly or tacitly condoned the other’s fault, he or she has the right to sever the common conjugal life, provided he or she has not consented to the adultery, nor been the cause of it, nor also committed adultery. (1983 CIC, c. 1152 §1)
"A spouse who occasions grave danger of soul or body to the other or to the children, or otherwise makes the common life unduly difficult, provides the other spouse with a reason to leave, either by a decree of the local Ordinary or, if there is danger in delay, even on his or her own authority. In all cases, when the reason for separation ceases, the common conjugal life is to be restored, unless otherwise provided by ecclesiastical authority. (1983 CIC, c. 1153 §1, §2 ).
If you desire to remain faithful to your Catholic obligations, unless there is “danger in delay,” even given lawful reasons to separate from your spouse as described above, you still need to bring a case for separation to the competent ecclesiastical authority.