When you say “OM…” I assume you’re referring to OMG.
Are you referring to them using the acronym or pronouncing the words?
I think many people neglect the use of the acronym OMG to mean Oh My Gosh or Oh My Goodness. I certainly used to use it like that.
As to them fully and utterly taking the name of the Lord in vain, I don’t think it’s mortal sin to not correct them. If I tried to correct everyone in my vicinity every time they swore like that, I would probably end up alone and crying in a corner without any friends in the world.
Also, it is possible for swearing like that to be a venial sin, especially if you know that they do so habitually.
If your family members are Catholic, then it wouldn’t hurt to try and remind them that it’s not a good idea to swear like that. But the thing is, if they know that it is a mortal sin, then they are probably working on breaking the habit already, and your reminders could end up just being constant annoying reminders of what they already know is flawed. If they don’t think it is a mortal sin, then you will not make much of a difference by correcting them over and over.
So I don’t think it is a mortal sin. If you really want to, you could correct them a few times, but I would not be constantly pestering them about it.