From the OP, as others have said, focus on "but certainly am glad to give you some time to discuss your situation." Take that opportunity to ask him what he meant, and to give him a chance to get to know you, which may also help you with your discernment. Sometimes it takes the objective perspective of someone else to help us to come to understand ourselves better. Others often see things - both strengths and weaknesses - in us that we do not, or choose not, to see in ourselves. My Spiritual Director does that for me. I respect him for telling me what I need to hear about myself. And, no, it is not always easy for me to hear the truth about myself. But, I think - I hope - I grow a little every time. (He lets me know!) Continue to pray about it. Go - with an open mind!! If you go defensively, you will only hear what you predispose yourself to hear. Let it be a learning experience. Don't hesitate to ask any questions you may have. Are you really going to be at peace if you don't go??
I meet with my Spiritual Director on Tuesday. I look forward to it, but, I also know his honesty may throw me a curve I don't expect. It is good to be challenged to be better and to do better. I have no doubt he has the best interests of my soul at heart.
At every Mass we say, "Lord I am not worthy to receive You". It doesn't end there, though, does it? "...but, only say the word, and I shall be healed."