How can you read the church fathers if you can't read the Bible?

A stock argument for Catholicism and Eastern Orthodox is that it's arrogant and naive to think you can read the Bible directly. An ancient book like Scripture is so far removed from the modern reader in space and time that we need the church fathers to interpret Scripture for us. 

The irony of this claim is that the same objection can be raised, perforce, to the church fathers. That, too, is ancient literature. They too are far removed from us in space and time. Different time, culture, and language. So that argument backfires.