I think it is less about them not knowing of his virginal birth and paternity, and more of them not understanding his message of detachment. Here we have a mission that Jesus is aware that he is on. He just let them see a glimpse of it, “I have to be detached from earthly things, even family ties, enough to be able to go do my father’s work. I can’t let anything hold me back.” How could they know his thinking? Know where he was going? For what he was sent for? They didn’t understand it then… and well, 2000 years later we are still trying to find words to express the entirety of that message.
The study bible I use most says:
**Did you not know: **Jesus is not rebuking Mary and Joseph, as though they had done something wrong, but instructs them on how their parental role must be subordinate to the will of his divine Father. His parents do have an important part to play in his mission, as indicated in the subsequent context, where Jesus submits himself to their leadership and honors them with the faithful obedience of a son (2:51). my Father’s house: Literally, “in that of my Father”. This could refer to the Temple specifically or to his mission from the Father more generally. - Ignatius Study Bible
Another good footnote for perusal:
2:41-52 Passover, which begins on the fifteenth day of the Jewish month of Nisan, is a feast that commemorates the freedom of the Jews from slavery in Egypt. At the time of Christ, all Jewish men were required to make a pilgrimage to the Temple in Jerusalem. The men often brought their families and traveled in caravans with other families, with men and woman traveling in separate groups and various children in both groups. It was within this arrangement that Christ’s absence from the caravan was not immediately noticed. Christ’s response at being found in the Temple indicated that he understood his identity as the Son of God and his mission to redeem humanity. This incident points to the Passion that would occur in the Holy City of Jerusalem. Joseph and Mary continued to accept their Son’s mission with humble faith. Christ’s three days in the Temple can also be seen as a prefiguration of his three days in the tomb before his Resurrection. (CCC 534, 583, 2599) - Didache Bible
And of course the standard NAB footnote:
2, 49: I must be in my Father’s house: this phrase can also be translated, “I must be about my Father’s work.” In either translation, Jesus refers to God as his Father. His divine sonship, and his obedience to his heavenly Father’s will, will take precedence over his ties to his family. - The Catholic Study Bible, 2nd Edition (NAB)