I’m only a nearly 21 year old kid with no children of my own, but I have two younger siblings and I’ve been through that teenager phase before. Here’s my advice as someone who can, at least somewhat, sympathize with your son.
My younger brother is, honestly, probably more intelligent than I am, and is a bit better than I am at music, despite my having taken piano since he was an infant. I tend to make better grades though because those are more important to me than they are to him. But he is more technically gifted with music, despite my studying it longer. My natural reaction is to get jealous and defensive of my own musical talent. It’s just an older sibling thing. I’ve been around longer so I should be better at this. It’s a pretty natural sibling rivalry thing.
In your son’s case, it seems like he’s acting out because his younger brother does better in school. On top of that, he probably feels like his sibling gets more attention than him, despite him being older, and on top of that has less strict social expectations without necessarily understanding why. Obviously I can’t speak for your son, but from the perspective of a fifteen year old (who you must remember do NOT have the sense of fairness and tolerance of an adult) it probably seems like his brother can “get away with more” than he can even though his brother is “smarter”.
To be completely frank, at his age, seriously bringing him in for an exorcism for being a rebellious teenager will more than likely turn him from his faith. I think that’s a horrible idea unless he is without a doubt possessed. All he will get from that is that his mom hates him and doesn’t understand, and that he can’t go to his parents and expect them to understand his problems. You need to be compassionate and understanding and at least try to understand his perspective or else he’ll just become resentful that his parents send him to his room when he acts out and then thinks he’s possessed because he doesn’t know how to express his problems without being punished.
Before you do anything else, you both need to talk to your son and try to hear his perspective. He may not want to talk to you about it, but you need to make it clear that you’re his parents, not the overbearing antagonists in his story. You have authority and you have your reasons for disciplining him. That doesn’t mean he understands, and if you react to his problems with what he sees as strict and excessive punishment, he won’t change his behavior. I have to go to mass, but I agree with the other posters. Just try to be understanding. I guarantee you this is a miscommunication issue, not demonic activity.