SHOULD be is not the same as MUST be.
He should be wearing the stole, technically.
However, a Confession is not invalidated because of him not wearing a stole. As noted above, he could be wearing beach clothes and an AC/DC T-shirt and it would still be valid.
Now, if he tried to absolve you by saying “I pray God forgives you, and have a blessed day”, then the Confession would be invalid… but so long as the necessary form is said “I absolve you [of your sins in the Name of the Father, and of the Son, and of the Holy Spirit]”, then the Confession remains valid.