Take this song, and assume the “her” being referenced is the vocation that one is responding to, and that gives you a pretty good idea of my experience: https://www.youtube.com/watch?v=b0LNhIaGAUw
Your spiritual director is an important resource, but ultimately he can only help you see your own conclusions - his job is primarily to help you identify how the Holy Spirit is working in your life, any patterns that might exist, and how to increase the consolation through your own responses.
…in the seminary, he’s also (effectively) the one guy you can talk to without worrying about it being used against you.
I’ll say this: the vast majority of guys do not have an easy time on the day-to-day, or even the longer term. Ease and happiness are not necessarily indicative of a vocation (though they do help, eh doc?); rather, does one find themselves being fulfilled by the ministry? The vocation you are currently responding to is not to be a seminarian (although you can’t ignore your current reality), but rather to be a priest.
My personal vote is you give it a summer assignment before making a decision. That being said, I do know of one friend who had a similar situation where things simply clicked and it was so obvious to him that he was called elsewhere; but again, this was after a particularly frank self-assessment of his summer and who he was in responding to the challenges and opportunities.
Most guys who are here with me would be happier as a husband and father, it’s just human nature… But again, that’s just the (unfortunate) reality of the seminary.