Early Church Father St. Jerome "defends the mutual perpetual virginity of Joseph and Mary:
The great doctor of the Church, St. Jerome (420 A.D.) is the outstanding defender of Joseph against the apocrypha and their early marriage, and states emphatically, "Certain people who follow the ravings of the apocrypha fancy that the brethren of the Lord are sons of Joseph from another wife…W e understand the brethren of the Lord NOT as sons of Joseph but the cousins of the Savior, children of Mary (the Lord's maternal aunt) who is said to be the mother of James the Less and Joseph and Jude…indeed, all Scripture indicates that cousins are called brethren."
Against the heretical interpretation of the text, "An He did not know her till she brought forth her firstborn son" (Mt.1:25), Jerome staunchly defends the mutual perpetual virginity of Joseph and Mary. Jerome states, "…it is customary in holy Scripture to call "firstborn" not him from whom brothers follow, but him who is first begotten. (Jerome, In Mt.1,25 (Pl 26:25). Likewise, the word "till" denotes a state or action up to a certain point, but does not necessarily denote a change thereafter, as in St. Paul's quotation from Psalm 109:1, "For He must reign 'until' He has put all enemies under His feet.'" (1 Cor.15:25)
And now, in one of the most beautiful tributes to St. Joseph to be found anywhere in the literature of the Church, Jerome proceeds to affirm that Joseph, like Mary, was perpetually virginal: "…we can contend that Joseph had several wives because Abraham and Jacob had several wives, and that from these wives the brethren of the Lord were born - a fiction which most people invent with not so much pious as presumptuous audacity. You say that Mary did not remain a virgin; even more do I claim that Joseph also was virginal through Mary, in order that from a virginal marriage a virginal son might be born. For if the charge of fornication does not fall on this holy man, and if it is not written that he had another wife, and if he was more a protector than a husband of Mary, whom he was thought to have as his wife, it remains to assert that he who merited to be called the father of the Lord remained virginal with her." (Jerome against Helvidius, 10-PL23:203). Jerome perceived that there were no historical reasons for supporting the tale of Joseph's wife and children; therefore, he constructed this magnificent argument on grounds of so-called congruity or propriety. This argument doesn't lead to certainty, but it is highly probable, and can be relied on for more than the manufactured legend which it opposes. "Either Joseph had other wives, or he committed adultery outside of marriage, or he lived virginally with Mary. But the idea of other wives can be dismissed because its sole claim to fame is its origin in an unreliable source; the charge of adultery can be rejected because he was a holy man; therefore, we can accept only the final possibility, namely, that Joseph lived virginally with the Mother of God, as her husband. Also, for Joseph to have been a widower with several children, it would have been necessary for him to have reached at least middle age when he took Mary into his keeping; but all the evidence for his marriage to Our Lady at the normal age would show that he was not old enough to have raised such a family." [https://www.catholicculture.org/culture/library/view.cfm?recnum=9298]