It can be devastating, depending on the relationship between the minister and the congregation. If the minister is leaving under duress, for example, sometimes it will split the church into supporters and detractors, and that particular congregation may fail.
If the minister was popular and especially if he or she had been there for a long time, bringing in someone new can also split the congregation or create hardship for the new minister. This, I believe, is one of the reasons behind the practice that I believe the Methodists and Lutherans used to have (don’t know if it is true anymore) of rotating ministers every few years.
In some of the nondenominational megachurches the entire church is built around a charismatic powerful leader and when that person leaves or dies (or, as sometimes happens, is found to have feet of clay), perhaps after decades, the church collapses. There is no larger organizational structure to support and assist it and the whole thing was tied up in a “cult of personality” built around this particular minister. This can happen even if there are multiple assistant ministers who may step forward, even if the replacement is the son of the founding minister.
Our own church (Unitarian Universalist) is in the early stages of replacing our minister as he is retiring. He has been with the church for 15+ years, so it is hard to imagine it without him, but he has given us lead time to make the process gradual and a chance to make it a positive experience. He will leave at the end of the next fiscal year, then the rules of the UU Association require that we have an interim minister for a year before we can have another settled minister, to give time to break between the two and start with a “clean slate,” if you will. The policy is beneficial, so that there is less of a burden on the new minister of expectations that he or she will be exactly like the old one.