I like your question, because this is EXACTLY what every single person considering a call to religious life (or the priesthood) wrestles with every day. EVERY DAY. “How do I know I’m called?”, “How do I know i’m not called to married life?”, “how do I know I’m doing the right thing?”, “am I doing the right thing?”,…but also “how do i know this anxiety is normal apprehension and not God telling me to forget it?”
But… to be perfectly honest, these are not easy questions to answer. And I don’t think I can adequately answer it at all. That’s why those discerning a vocation, religious formation programs, and seminaries devote a LOT of resources toward spiritual direction. This is what a spiritual director should hopefully be good at, helping you to sort through those different feelings, emotions, anxieties, and determine what is minor, and what is coming from somewhere deeper.
As you said, marriage is an attractive vocation, and religious don’t apply to join because they don’t want to be married, or they aren’t attracted to the opposite sex. You’re absolutely right in that it’s a sacrifice. But on the other hand, you have to be attracted to religious life as well. If you completely dislike the idea of being a religious or a priest, then you don’t have a vocation (at least not at that time…God works in mysterious ways and can absolutely change your mind. I know this first hand). So religious life has to have some element of attraction to you That level of attraction may diminish, grow or stay the same over time. If it fizzles out, you probably weren’t being called to that vocation, if it gets stronger over time or “just won’t go away”, then you need to take a deep hard look into that, start with your pastor, then contact a vocation director, get a spiritual director. You still may discern out of a religious vocation, maybe even years down the road, but it’s definitely something you should look into if you’ve had a persistent attraction to it.
Also, everyone’s story is different. No two stories of discernment or vocation journey are the same, which also makes this question hard to answer. Some people felt called late in life, some people have known since they were 3, some knew instantly, for others it was a gradual process. For me, it was a gradual process, but in hindsight, I can see looking back that I’ve “known” for most of my life, although I would never have admitted it then, and in fact tried to runaway from it in a sense. I’ll also tell you the most difficult thing I’ve done to date was break up with my girlfriend of several years in order to pursue religious life. I had an attraction to marriage which honestly seemed inevitable, but I also had an attraction to religious life for quite some time. Through prayer, I finally got the courage to talk with someone about it (my pastor initially), meet with various vocation directors, work with a spiritual director, and spend time with different communities to learn about them, and once I spent time with the “right” community I just knew it was what i had to do (although not surprisingly it was the same community that spawned my interest in religious life over 10 years before).
I had an attraction to religious life, but at the same time I didn’t want to at all because I didn’t want to end the relationship I was in, so even though i “knew” deep down, I still didn’t want to acknowledge it. It took another year before I broke the news to my girlfriend, ending that relationship, and began the application process, then formation.
I don’t know how else to explain it other than tell you the little about my story that i just did. If you talk to other men and women discerning their vocation, get their stories too. They’ll all be different, but you’ll start to see similar elements begin to emerge, you’ll begin to see how different people were able to “draw the line” as you put it, to figure out if it truly was a vocational call or just a passing interest. You’ll be able to see how your own story is starting to play out as well.
I know someone else who also ended a relationship to apply to religious life, but he couldn’t even bring himself to complete the application. He knew pretty quickly that it just wasn’t right… he is now getting married, and couldn’t be happier; different vocation than he thought, but I know he’ll be a better husband and father because of it. So the answer to your question is there, but it isn’t very clear. It’s not a clear line. You can’t say “if the signs are XYZ then it’s married life, or ABC religious life”, or “feelings 1,2, and 3 are from God, but 4, 5, and 7 is my own petty nervousness”. It’s a personal process, and learning to distinguish these things takes time, prayer, and it definitely takes experiencing it. It’s helpful to have a guide too (i.e. spiritual director!).
I’m only in early formation, and haven’t even made first vows yet, so my story is still being played out…but I will say I’m very happy with my decision so far.
Ultimately though, we just have to trust God. He’ll let you know. He is VERY generous.