An excursus in one of Steve Hays' e-books, This Joyful Eastertide, addresses some early pagan responses to Christianity. He discusses the arguments of four early critics: Galen, Celsus, Porphyry, and Julian the Apostate.
I've written about early non-Christian support for much of what Christians believe. For example:
The Slaughter Of The Innocents
The Darkness At Jesus' Crucifixion
The Empty Tomb
How The Early Christians Viewed The Resurrection
The Historical Genre Of The New Testament Documents And Christian Interest In Historical Evidence
The Death Of The Apostles
Christian Moral Standards And Practice
Christian Opposition To The Veneration Of Images
Christian Rejection Of Prayer To The Dead And Angels
That Mary Wasn't A Perpetual Virgin After Jesus' Birth
The Significance Of The Gospels
New Testament Authorship
The Old Testament Canon
The New Testament Canon
The Text Of The New Testament
The post here addresses a large series of issues related to what set Christianity apart in the ancient world. The posts linked there are based on a book by a New Testament scholar that, among other things, discusses what ancient non-Christians thought of the religion.
Those are just several examples. There are more that can be found in the archives, and a lot of other relevant material hasn't even been discussed at this blog.
One means by which people often attempt to dismiss the evidence from non-Christian sources is by arguing that the early enemies of Christianity were apathetic about the religion. They went along with some of the false claims Christians were making rather than take the effort to argue against those claims. Or they didn't give the religion much attention until it was too late or too difficult to discern and prove the falsity of the Christian claims. I address such arguments in some of the material I've linked above, and here's a post I wrote that focuses on the apathy issue. Here's one that addresses the argument in the context of the empty tomb.
Another common response is to suggest that the early Christians may have suppressed some better arguments against Christianity, so that those better arguments are no longer extant. Maybe these Christians were lying or mistaken about what their opponents were saying. See the thread here, including the comments section, for a discussion of some of the problems with such theories.
The significance of non-Christian corroboration varies from case to case. I only assign a little significance to the non-Christian testimony concerning the Slaughter of the Innocents, for example, since the sources are so vague (Assumption of Moses) or so late and garbled (Macrobius), among other problems. On the other hand, Jewish acknowledgment of the empty tomb seems to have been early, clear, widespread, and persistent, concerning a matter they were in a good position to judge.
Factors like non-Christian apathy and Christian dishonesty and selectivity, as well as honest misunderstandings on both sides, surely played a role. The question is the extent to which they did so. For reasons I explain in some of the material linked above, the context in which Christians and non-Christians agreed on issues like Jesus' birth in Bethlehem and the empty tomb makes it much more likely that they agreed about the truth than that they agreed about an error. The degree to which modern critics have to suggest that both the early Christians and their opponents were widely or universally wrong should give them pause.