This passage from the old Catholic Encyclopedia may help you.
“Turning now to the historical development of the present law of celibacy, we must necessarily begin with St. Paul’s direction (I Tim., iii, 2, 12, and Titus, i, 6) that a bishop or a deacon should be “the husband of one wife”. These passages seem fatal to any contention that celibacy was made obligatory upon the clergy from the beginning, but on the other hand, the Apostle’s desire that other men might be as himself (I Cor., vii, 7-8), already quoted) precludes the inference that he wished all ministers of the Gospel to be married. The words beyond doubt mean that the fitting candidate was a man, who, amongst other qualities which St. Paul enunciates as likely to make his authority respected, possessed also such stability of divorce, by remaining faithful to one wife. The direction is therefore restrictive, no injunctive; it excludes men who have married more than once, but it does not impose marriage as a necessary condition. This freedom of choice seems to have lasted during the whole of what we may call, with Vacandard, the first period of the Church’s legislation, i.e. down to about the time of Constantine and the Council of Nicaea.”