It’s about jurisdiction.
Absolution is a juridic act of the Church. In other words, it is a legal act. So, just like a sheriff or a magistrate has jurisdiction, so too do pastors.
Just as an aside, in ancient Rome, at the end of a trial, a jury did not vote “guilty or innocent” (like we do today) but rather “condemno or absolvo” (obviously, condemn or absolve). Confession is a court. The priest is the judge. Thankfully, by the Grace of God, the verdict is usually “absolvo.”
Also, the Catholic Church operates primarily on the diocesan level—this is not just practical but theological, even though most people might not always think that way. Local parishes are not just “branch offices of the Vatican” like we might see in the business world, nor are we just a collection of individual locales like we might see in some Protestant models (like the loosely affiliated Baptist conventions). Note to readers: see Vatican II document Lumen Gentium Chapter III.
Again, because absolution is a juridic act, a priest must have the juridic authority to make such an act valid. It isn’t enough that he be validly ordained, but he must have the authority to act in the name of the Church. Remember that he says, towards the end of the absolution “through the ministry of the Church…I absolve you…” In order to truly be a minister of the Church, he must be able to speak with the authority of the Church.
It is worth noting here that one of the changes made from the 1917 Code of Canon Law to the 1983 Code is this:
967 §2. Those who possess the faculty of hearing confessions habitually whether by virtue of office or by virtue of the grant of an ordinary of the place of incardination or of the place in which they have a domicile can exercise that faculty everywhere unless the local ordinary has denied it in a particular case…
In strictly practical terms, that means that most priests do indeed have worldwide faculties to absolve while the fact remains that every priest still must have such faculties in order to validly absolve.