Even celibate priests and nuns who do full time mission/ministry take time to be with their communities and most of the time have separate quarters for their communities. A family who’s involved in full time ministry should do the same.
The chaplain for an interdenom Christian group I’m part of is also an AG pastor and as devoted as he is to his ministry towards others, his first duty is towards his family. If one of his kids is sick, I don’t expect to see him in his office. Similarly, our priest is a Dominican priest who’s dedicated to his ministry but at the same time, his first duty is towards his order, especially since he was sent here through them (on orders from our bishop). Regardless of your vocation, your first priority is the vows you took, whether as husband and wife, as an ordained priest/deacon, or as a religious. Nothing should contradict that.
And just because I’m a woman doesn’t mean I’m being soft on women. If this was a female pastor, I’d give the same answer.