You are on the right track,I would say. Sex is a big deal, and it should be left to married people.
Unfortunately you are going to have a hard time finding someone who feels the same. I can absolutely assure you of that. Even the Catholics you see in the Church pews Sunday after Sunday going up for communion are doing it. Because the issue of sex before marriage just isn't dealt with properly by Catholics, and for such a time as this when the secular world is obsessed with sex! Satan really got his way this time around :(
But you shouldn't be too obsessed with your man being a physical virgin, but a spiritual one yes. Guys and girls make mistakes. I myself have made the mistake, with one girl, and wish I could take it back. But I can't so I confessed am forgiven and now have to remain pure for my future wife! And in all ways, I don't masterbate and I avert my eyes at swimsuit magazines and even women who are dressed immodestly (that kinda thing is unattractive to me anyway). Also I personally wouldn't date someone who has had many sexual partners and then claims they have realized it is wrong. Personally, I am not good natured enough to get past it because I do believe a mistake is a mistake and made repeatedly it reveals something more (i.e. how many times can you sincerely call a mistake a mistake).
Anyway, I would say just relax. I am trying myself, too. It is hard, because it seems to me that it is impossible to find a girl with the same values as me. I mean really, I know ALL TYPES of girls and they are all sexually active. And it makes me panic at times. Because you're right, sex takes a part of your soul and gives it to another - whether you like it or not that is what happens and that is what is meant to happen! But people see nothing wrong with it now. My ex girlfriend for example, she told me she doesn't regret sex with her ex's because it was the right thing to do at the time and she will always love them in some way. Now... that kind of thing is STRAIGHT from Hollywood; forever romanticizing it all. But that kind of thing also has no place in marriage or in a relationship, because once again your soul and heart should completely belong to one person and not anybody else.
And the saddest thing is that I have conversations all the time with good girl friends of mine, and I tell them why waiting is the right thing to do and I can speak for 15 minutes and really make them smile and I can see they feel something from it - because they recognize the truth and real love. But they go away that night and still sleep with a random stranger. How can we get through to people... But I have hope. I pray, and focus on my own purity. It is so hard in this secular world. And it is almost expected of you to sleep with people. Personally I'm pretty good looking, so girls do come up to me a lot and they offer me things, some are even straight to the point and I have had a girl tell me I can take her anywhere I want and she will let me do anything to her. I am usually gobsmacked and decline politely, and I have been slapped before because they think its rude to decline being offered because hey, sex just has to be had!
Pray to God and focus on yourself and other virtues, and the right man will come!