Scripture only positively affirms that the Virgin Mary gave birth to Jesus. Scripture identifies some people as the brothers and sisters of Jesus but whether or not those people are the biological children of the Virgin Mary is not stated. So, the pastor went too far when he said that the Virgin Mary didn’t remain a virgin and that she and Joseph had many children, if he meant she gave birth to other children besides Jesus.
Scripture says, Joseph “took his wife [Mary], but knew her not until she had borne a son; and he called his name Jesus.” (Matthew 1:24) This Scripture only positively affirms that Joseph and the Virgin Mary did not have marital relations before the birth and naming of Jesus. Although the word “until” is often used to imply a change of circumstances afterwards, that is not always the case. Thus, the use of the word “until” in this Scripture could imply that Joseph had marital relations with the Virgin Mary after Jesus was born and named but not necessarily.
There is a very early Christian tradition, described in a 2nd-century work sometimes referred to as the Protoevangelium of James, that Joseph was an old widower with children, including at least two sons, from his previous marriage, when he married the Virgin Mary. Those children of Joseph by his previous marriage could well account for the so-called brothers and sisters of Jesus mentioned in Scripture.