As you noted, the Roman Missal (and its General Instruction) direct the priest to face the altar at times, and face the people at times. That assumes ad orientem worship; that is, worship facing “the east”.
The priest is instructed to face the people for the Greeting (GIRM 124), the “Pray, brethren, that…” (GIRM 146), the offering of the peace (GIRM 154), the “This is the Lamb of God…” (GIRM 157), the prayer after Communion (GIRM 165), and the final blessing and dismissal. (GIRM 185)
That the priest is directed to face, or to turn toward, the people, implies that he has been facing something other than the people. This is particularly true during the whole Liturgy of the Eucharist; he turns to face the people during the “Pray, brethren…”, and then can face the altar until the offering of the peace, and then he faces the altar again until he shows the Host to the people saying, “This is the Lamb of God…”