The line becomes blurred, it is not always clear, where we pass from having a thought occur to us, to willfully engaging in this thought. Drive them from your mind and pray as soon as you are able to. Sometimes you need to pray something continuously, something simple like the Jesus prayer of our Eastern friends (Lord Jesus Christ, son of God, have mercy on me, the sinner) so that you can avoid such thoughts.
Actively engaging your will, even for a moment, to wish harm upon another is a sin. Whoever thinks lustfully about a woman has committed adultery in his heart. Again, are you perfectly knowledgeable of exactly when you are engaging your will in a thought? God, however, knows all that is in our hearts.