Chromatius of Aquileia, who lived in the fourth and fifth centuries, wrote:
"But concerning what the evangelist said, 'And he did not know her till she brought forth a son' [Matt 1:25], several foolish people are accustomed to stir up a question, thinking that after the Lord's birth saint Mary was united with Joseph." (Thomas Scheck, trans., Chromatius Of Aquileia: Sermons And Tractates On Matthew [Mahwah, New Jersey: The Newman Press, 2018], approximate Kindle location 3051)
Though Chromatius could be discussing opponents of Mary's perpetual virginity in general, he seems to be limiting his comments to the interpretation of Matthew 1:25 instead. Either way, his use of the term "several" is significant. There surely were some people who held the view in question with whom Chromatius wasn't familiar. So, the total number has to be higher than the several Chromatius refers to. And if he's only commenting on a particular interpretation of Matthew 1:25, then the total number who rejected the perpetual virginity of Mary, whether on the basis of Matthew 1:25 or on other grounds, must have been higher still. Advocates of the perpetual virginity of Mary often say or suggest that only one or two individuals or some other extremely small number denied her perpetual virginity before the Reformation (only Helvidius, only Tertullian and Helvidius, etc.). Chromatius' comment suggests the number was higher.
And we have far more than Chromatius' comment to go by. See here and here, for example for discussions of the evidence that many individuals rejected Mary's perpetual virginity for hundreds of years before the Reformation, beginning in the first century and continuing into the medieval era. Rejection of her perpetual virginity seems to have been the more popular view during the earliest generations of church history.
