Jason, where would you recommend someone start with the church fathers, both in terms of primary and secondary reading? It seems such a dauntingly large field to a non-specialist...
I'm going to broaden the subject to the study of the fathers in general, not just primary and secondary reading. I'll include an answer to your question, but I'll also address some other issues along the way.
A lot depends on your interests and the time and other resources you have. If you're more interested in soteriology than ecclesiology, then you should take a different approach than if you were more interested in ecclesiology. If you do most of your apologetic work on Mormonism, then the approach you should take will be different than if you're more focused on atheism. I'm an apologist, and my reading of the fathers is largely on apologetic issues, so my patristic interests and knowledge reflect that. By its nature, apologetic work will tend toward earlier sources. I'm more familiar with the earlier fathers than the later ones. Examine your interests and resources, and make your plans accordingly.
The more familiar you are with the arguments of Muslims, Jehovah's Witnesses, Roman Catholics, and other groups, the better you'll be able to notice the relevant material in the fathers. The more knowledgeable you are of the relevant issues before you read the fathers, the more you'll get out of them. Take some time to think about what you want to look for in the fathers before you begin studying them.
You could read the fathers as you would anything else, maybe reading dozens of pages one day, but nothing or only a couple of pages another day. Or you could establish a practice of reading from the fathers daily. Even if you just read two pages a day, that will amount to a lot over time. I do both. I read from the fathers daily, but I also read beyond that daily pattern when I have the time and interest.
As you study the fathers and related issues, take a lot of notes, and keep them well organized. Don't just take notes on the issues you're most interested in. Note anything of significance, whatever the topic. It's helpful to have extensive notes for future reference.
Regarding which documents to read and in what order, I'll make some general recommendations. You can adjust these recommendations according to your circumstances.
Start with the earliest fathers. Earlier sources are helpful in making sense of later sources, the earlier sources have more evidential value due to their earliness, and the brevity of their extant writings makes them an easier place to start. Begin with Clement of Rome. Move on to Polycarp, Papias, Ignatius, and The Letter To Diognetus. If you aren't going to read all of the earliest fathers, or you're going to move on to some later fathers before returning to the earliest ones, some of the earliest sources that you can most easily avoid are The Epistle Of Barnabas and The Shepherd Of Hermas. Hermas, for example, is relatively lengthy and unprofitable.
Among the later fathers, Justin Martyr's Dialogue With Trypho would be a good place to begin. Irenaeus' Against Heresies is important, but it's lengthy, and the first two books are difficult to get through. He goes into a lot of depth about the beliefs of heretics, and you'll agree with him that "Even to give an account of them is a tedious affair, as you see." (Against Heresies, 1:31:4) I recommend reading the whole work, but if you aren't going to do so, then at least read the third book. Also read Origen's Against Celsus. Eusebius' Church History is important as well. John Chrysostom's homilies are often eloquent and insightful. I'll stop there, but a lot more could be added. There's enough to last a lifetime.
In the context of apologetics, the most important documents are the Papias fragments, Justin's Dialogue, Irenaeus' Against Heresies, Origen's Against Celsus, and the Church History of Eusebius. The most significant of those is Against Celsus. It's available for free online, but I recommend reading Henry Chadwick's edition, Origen: Contra Celsum (New York, New York: Cambridge University Press, 2003). It's a more recent translation, and there's a lot of valuable material in his introduction and notes.
Which modern sources on the fathers you should read depends on your interests. I'll use Origen's Against Celsus as an example, since I just mentioned it. If you get Chadwick's edition, you'll get his introductory comments and notes along the way. You could supplement that with something like John McGuckin's Westminster Handbook To Origen (Louisville, Kentucky: Westminster John Knox Press, 2004). But for general, introductory purposes, a good source to start with is Everett Ferguson, ed., Encyclopedia Of Early Christianity (New York: Garland Publishing, Inc., 1999).
You can read some old translations of many of the fathers for free online at ccel.org. A version that's easier to navigate, but without notes and other supplementary material, can be found here. Roger Pearse has a collection of other patristic documents and an entire site dedicated to Tertullian. He also has a good blog that often addresses patristic issues. Tom Schmidt has some good resources at his blog. He has the best collection of Papias fragments I've seen. See the sidebar of his blog for links to other patristic resources. You can also find many relevant posts in the archives of this blog. I've posted many book and article recommendations on patristic issues, articles about patristic theology, etc.
Here's how I'd recommend getting started. Get, for reference purposes, Everett Ferguson's Encyclopedia Of Early Christianity (New York: Garland Publishing, Inc., 1999). Get Michael Holmes' The Apostolic Fathers (Grand Rapids, Michigan: Baker Books, 2005). Read his introduction to First Clement, read my article on the identity of the document's author here (scroll down for it), then read First Clement itself. Read Holmes' introduction to Polycarp's Letter To The Philippians, read my article on Polycarp here, then read Polycarp's letter in Holmes. Next, read Holmes' introduction to Papias, read my article about him here, then read Tom Schmidt's collection of Papias fragments (not Holmes' collection). Read Holmes' introduction to Ignatius, read my article about him here, then read his letters in Holmes' book. Last, read Holmes' introduction to The Epistle To Diognetus, then read the document.
After that, you'll have to decide how you want to proceed. If you want to read more of the earliest fathers, I'd give priority to Aristides' Apology (available for free online, but not in Holmes) and The Martyrdom Of Polycarp. The Didache, Second Clement, The Epistle Of Barnabas, The Shepherd Of Hermas, and the fragment of Quadratus (available for free online and quoted and discussed only in passing in Holmes) should have the lowest priority. Among the later fathers, start with Justin's Dialogue With Trypho.