Joseph and Mary were betrothed at that time, but not fully married. Betrothal was a binding contract, but it was the period of engagement before marriage. During this time the couple does not live together nor do they have sexual relations. It is important because it must be clear that Jesus is NOT the offspring of Joseph.
Having been enlightened by an angel in a dream regarding her pregnancy, and perhaps further by Mary concerning the words of the archangel Gabriel to her at the Annunciation, Joseph knew that God had conducted himself as a husband in regard to Mary. She was now prohibited to him for all time, and for the sake of the Child and Mary he could only live with her in an absolutely chaste relationship.
Since Joseph and Mary never have intercourse there is NO question of adultery. A virgin cannot be an adulterer! Additionally, God would be incapable of the evil act of adultery.
St. Augustine (391-430AD) “a virgin conceiving, a virgin bearing, a virgin pregnant, a virgin bringing forth, a virgin perpetual.” #1518, III, p.30; also: #1642, III, p. 71; #1974d, p. 166.
St. Athanasius (358-362) in his Discourse Against the Arians called the Mother of Jesus, "the ever-virgin Mary" #767a, I, p. 330.
Didymus the Blind (381-392 - He was head of the catechetical school at Alexandria) “for neither did Mary. . . marry anyone, nor did she ever become the mother of anyone else, but even after childbirth she remained always and forever an immaculate virgin.” #1073, II, p. 62.
St. Epiphanius of Salamis (374) “the holy ever-virgin Mary” and “to Holy Mary is invariably added, for that Holy Woman remains undefiled.” #1111, II, p.76.