In “The Love of Solitude and Silence” he writes:
*SEEK a suitable time for leisure and meditate often on the favors of God. Leave curiosities alone.Read such matters as bring sorrow to the heart rather than occupation to the mind. If you withdraw yourself from unnecessary talking and idle running about, from listening to gossip and rumors, you will find enough time that is suitable for holy meditation.
Very many great saints avoided the company of men wherever possible and chose to serve God in retirement. “As often as I have been among men,” said one writer, “I have returned less a man.” We often find this to be true when we take part in long conversations. It is easier to be silent altogether than not to speak too much. To stay at home is easier than to be sufficiently on guard while away. Anyone, then, who aims to live the inner and spiritual life must go apart, with Jesus,from the crowd*
I don’t think he’s saying that everyone should live in solitude at all times. He’s only saying that many did.
Remember that the books are known collectively as “The Imitation of Christ”. Jesus had solitary times - notable in the Garden of Gethsemane - but was often surrounded by crowds.
But really I don’t think there’s one size fits all. We all need to serve God in our own ways and according to our own abilities.