Because they were going in different directions.
Tradition has it that the priest and Levite were going from Jericho to Jerusalem.
The man who was attacked was going from Jerusalem to Jericho.
The Samaritan wouldn’t have had business in Jerusalem, so presumably, he was headed elsewhere.
The point of the parable is this: the priest and Levite were concerned only with ritual purity laws (they didn’t want to become ritually impure and therefore, unable to perform their duties), whereas the Samaritan (who didn’t have to worry about Jewish ritual purity) was a better neighbor to the injured man.
In any case, even if they were all going in the same direction, the priest and Levite wouldn’t have traveled with the Samaritan; presumably, that close an interaction would risk ritual impurity, too…