One of the reasons for rejecting the papacy is the lack of justification for it. There are apparent contradictions of the concept of the papacy in some New Testament documents and other early sources, but the lack of evidence for the office would be enough reason to not accept it, even if such contradictions didn't exist.
However, Protestants often focus on too narrow a range of contexts in which the papacy is absent in the early sources. A lot of attention is given to passages about Peter in the gospels and Acts and material about church government in the early sources, for example, but we ought to think more broadly about where a papacy could have been mentioned if it existed. A papacy wouldn't have to be mentioned at every conceivable opportunity. But the larger the number and variety of contexts in which a papacy could have been mentioned, but wasn't, the more likely it is that the office didn't exist. What I want to do in this post is provide a few examples of contexts that are often neglected.
The apostles sometimes discussed their upcoming death, what was being done to preserve their teachings, and how Christians should conduct themselves going forward (e.g., Acts 20:17-38, 2 Timothy 3:10-4:8, 2 Peter 1:12-21). If the papacy was considered the foundation of the church, the infallible center of Christian unity throughout church history, the absence of any mention of such a resource in passages like these is significant.
Another group of relevant contexts is the imagery used to refer to relevant entities, such as what imagery is used to refer to the apostles or the church. We get twelve thrones without Peter's throne being differentiated (Matthew 19:28), three pillars without Peter's being differentiated (Galatians 2:9), twelve foundation stones without Peter's being differentiated (Ephesians 2:20, Revelation 21:14), etc.
The early Christians often interact with the objections of their opponents. The gospels respond to the charge that Jesus performed miracles by the power of Satan, Paul responds to his critics in his letters, Justin Martyr wrote a response to Jewish arguments against Christianity, Origen wrote a response to Celsus' anti-Christian treatise, and so on. See
here regarding the lack of reference to a papacy in such contexts.
It's important for Protestants (and other opponents of the papacy) to bring up considerations like these, since the absence of early references to a papacy becomes more significant when the absence occurs across a broader range of contexts. If only two pages of early Christian literature were extant, the absence of a papacy (or whatever other concept) would be much less significant than its absence across two
million pages. The number of pages matters (assuming the usual diversity of topics you'd get with an increase in such a page number).
One of the reasons why it's become so popular for Catholics to argue for the papacy by an appeal to something like typology or Old Testament precedent is that there's such a lack of evidence in the New Testament and the early patristic literature. So, there's a turn to other sources to try to find what isn't present where we'd most expect to see it.