The apostle Peter had many roles. He was a Christian. He was one of the Twelve, the original disciples of Jesus. He was one of the apostles (a group not limited to the Twelve). He was a spokesman for Christianity (e.g., Acts 2). According to some later traditions, he was a bishop of Rome and/or Antioch. He was a martyr. Etc.
When a church father or some other source refers to somebody as a successor of Peter, we ought to ask in what sense that individual is a successor. A person can succeed Peter in one sense, but not in another.
This principle is illustrated in one of the earliest extant commentaries on Matthew 16:
“And if we too have said like Peter, 'You are the Christ, the Son of the living God,' not as if flesh and blood had revealed it unto us, but by light from the Father in heaven having shone in our heart, we become a Peter, and to us there might be said by the Word, 'You are Peter,' etc. For a rock is every disciple of Christ of whom those drank who drank of the spiritual rock which followed them, and upon every such rock is built every word of the church, and the polity in accordance with it; for in each of the perfect, who have the combination of words and deeds and thoughts which fill up the blessedness, is the church built by God.” (Origen, Commentary On Matthew, 12:10)
Is Origen saying that all Christians are Popes? No. He’s saying that all Christians are what Peter was in a particular context. A person can succeed to Peter, or be what Peter was, in one sense, but not another. Somebody who succeeds Peter as a Christian doesn’t necessarily succeed him as an apostle, bishop, martyr, etc.
Keep such distinctions in mind when you see Catholics citing patristic passages that refer to the bishop of Rome as a successor of Peter. A successor in what sense?