Decoding Official Secrecy: Computational Analysis of Hundreds of Thousands of Declassified Documents

---Note: This event is co-sponsored by the Stanford Department of History---

The scope of official secrecy is rapidly expanding. The sheer scale of the national security state, the growth of electronic media, and the power that still comes from compartmentalizing information means that the government is only releasing a tenth as many pages of classified information as it produces. Hundreds of millions of secret documents are piling up, raising doubts about how we will be able to reconstruct the past and ensure government accountability. But historians are now teaming up with data scientists to analyze the millions of documents that are being released. Since these were among the first official documents produced and stored on computers, we can use techniques like natural language processing and machine-learning. It may now be possible to make out the broad patterns of official secrecy, attribute authorship to anonymous documents, and perhaps even predict the content of redacted text. But the political and ethical questions remain: what does the public need to know, and when do they need to know it?

About the speaker: Matthew Connelly is professor of history at Columbia University. His publications include A Diplomatic Revolution: Algeria's Fight for Independence and the Origins of the Post-Cold War Era (2002), and Fatal Misconception: The Struggle to Control World Population (2008). He has written research articles in Comparative Studies in Society and History, The International Journal of Middle East Studies, The American Historical Review, The Review francaise d'histoire d'Outre-mer, and Past & Present. He has also published commentary on international affairs in The Atlantic Monthly, The Wilson Quarterly, and The National Interest. He directs the University Seminar on Big Data and Digital Scholarship, the dual masters program with the LSE in International and World History, and the Hertog Global Strategy Initiative, a research program on the history and future of planetary threats. He received his B.A. from Columbia (1990) and his Ph.D. from Yale (1998).