Is it a mortal sin to use marijuana casually (once or twice a year)?
Drugs may only be taken for genuine therapeutic purposes. When taken for recreational purposes, which is implied by the qualifier casually, it constitutes grave matter:
The use of drugs inflicts very grave damage on human health and life. Their use, except on strictly therapeutic grounds, is a grave offense. Clandestine production of and trafficking in drugs are scandalous practices. They constitute direct cooperation in evil, since they encourage people to practices gravely contrary to the moral law (CCC 2291).
For there to be mortal sin, there must also be full knowledge and full and free consent to the grave action.