As noted by the previous poster, Pope Pius XII in the encylical Humani Generis states the following:
- When, however, there is question of another conjectural opinion, namely polygenism, the children of the Church by no means enjoy such liberty. For the faithful cannot embrace that opinion which maintains that either after Adam there existed on this earth true men who did not take their origin through natural generation from him as from the first parent of all, or that Adam represents a certain number of first parents. Now it is in no way apparent how such an opinion can be reconciled with that which the sources of revealed truth and the documents of the Teaching Authority of the Church propose with regard to original sin, which proceeds from a sin actually committed by an individual Adam and which, through generation, is passed on to all and is in everyone as his own.
The Pope’s statement reaffirms the de fide declaration of the Council of Trent that Adam’s sin is transmitted not by imitation, but by descent - by natural generation; and that all human beings inherit this condition as a result. Catholics are required to give the assent of faith to this revealed truth. That being said, does this necessarily mean that there must have been what we would call today “incestuous” relations between the offspring of our first parents?
I would agree with your priest that we are not compelled to believe this, although it may be true. This is where the discussion of monogensim and polygenism arises in connection with Pope Pius’ encyclical. It is imperative that we understand what the Pope means by “polygenism” here.
There are, in principle, two ways to view polygenism. First there is biological polygenism. That is where a common pool of hominids were interbreeding many thousands of years ago, resulting in varying degrees of genetic ancestry among modern homo sapiens. Second, there is metaphysical polygenism. It is not concerned about early the genetic ancestry of early hominids. It is concerned with human beings who have the metaphysical characteristic of the human soul.
It would certainly be heretical to claim that metaphysical polygenism is possible. That would mean that you could have two ultimate ancestors with human souls, say Adam and Eve, and I could have two completely different ultimate ancestors with human souls - say Bob and Claire. No, we must believe that we all have the same ultimate ancestor who was, metaphysically speaking, a human being with a soul.
Is Pope Pius here condemning biological polygenism or metaphysical polygenism, or both? It seems to me that he is not saying anything about biological polygenism. Whether or not there were early hominids (as opposed to human beings with a human soul) interbreeding for quite some time before Adam and Eve (the first two hominids with human souls) isn’t the issue. You will note that Pius refers to “true men” who took their origin from Adam ie. human beings with souls. He doesn’t say anything about earlier hominids without human souls. So it is metaphysical polygenism that is rejected by the Pope, not biological polygenism. A good article from Ed Feser summing up these distinctions can be found here: edwardfeser.blogspot.com/2011/09/modern-biology-and-original-sin-part-i.html
So what about the “incest” part? If all human beings today must have as a common ancestor Adam and Eve, it may seem like their children must have interbred… That conclusion isn’t entailed though. It is certainly possible that Adam and Eve’s children - say Cain - interbred with hominids who did not have human souls. Those children could have been instantiated with a soul at conception by God. They would have as a common ancestor our first parents, Adam and Eve, and would have inherited original sin as a result.
This is all speculative of course. It may be that Adam and Eve’s offspring only interbred with one another. But it is also possible that they didn’t. Some people would see this as bestiality and not any more preferable than incest. The point is though that your priest isn’t necessarily wrong in believing that there were no immediate interrelations between our first parent’s offspring.