Federal government statistics provide critical information to the country and serve a key role in a democracy. For decades, sample surveys with instruments carefully designed for particular data needs have been one of the primary methods for collecting data for federal statistics. However, the costs of conducting such surveys have been increasing while response rates have been declining, and many surveys are not able to fulfill growing demands for more timely information and for more detailed information at state and local levels. Innovations in Federal Statistics examines the opportunities and risks of using government administrative and private sector data sources to foster a paradigm shift in federal statistical programs that would combine diverse data sources in a secure manner to enhance federal statistics. This first publication of a two-part series discusses the challenges faced by the federal statistical system and the foundational elements needed for a new paradigm.

The environment for obtaining information and providing statistical data for policy makers and the public has changed significantly in the past decade, raising questions about the fundamental survey paradigm that underlies federal statistics. New data sources provide opportunities to develop a new paradigm that can improve timeliness, geographic or subpopulation detail, and statistical efficiency. It also has the potential to reduce the costs of producing federal statistics. The panel's first report described federal statistical agencies' current paradigm, which relies heavily on sample surveys for producing national statistics, and challenges agencies are facing; the legal frameworks and mechanisms for protecting the privacy and confidentiality of statistical data and for providing researchers access to data, and challenges to those frameworks and mechanisms; and statistical agencies access to alternative sources of data. The panel recommended a new approach for federal statistical programs that would combine diverse data sources from government and private sector sources and the creation of a new entity that would provide the foundational elements needed for this new approach, including legal authority to access data and protect privacy. This second of the panel's two reports builds on the analysis, conclusions, and recommendations in the first one. This report assesses alternative methods for implementing a new approach that would combine diverse data sources from government and private sector sources, including describing statistical models for combining data from multiple sources; examining statistical and computer science approaches that foster privacy protections; evaluating frameworks for assessing the quality and utility of alternative data sources; and various models for implementing the recommended new entity. Together, the two reports offer ideas and recommendations to help federal statistical agencies examine and evaluate data from alternative sources and then combine them as appropriate to provide the country with more timely, actionable, and useful information for policy makers, businesses, and individuals.

The past 50 years have witnessed a revolution in computing and related communications technologies. The contributions of industry and university researchers to this revolution are manifest; less widely recognized is the major role the federal government played in launching the computing revolution and sustaining its momentum. Funding a Revolution examines the history of computing since World War II to elucidate the federal government's role in funding computing research, supporting the education of computer scientists and engineers, and equipping university research labs. It reviews the economic rationale for government support of research, characterizes federal support for computing research, and summarizes key historical advances in which government-sponsored research played an important role. Funding a Revolution contains a series of case studies in relational databases, the Internet, theoretical computer science, artificial intelligence, and virtual reality that demonstrate the complex interactions among government, universities, and industry that have driven the field. It offers a series of lessons that identify factors contributing to the success of the nation's computing enterprise and the government's role within it.

On May 8, 2009, the symposium, The Federal Statistical System: Recognizing Its Contributions, Moving It Forward was held in Washington, DC. One of the topics considered at that symposium was the health of innovation in the federal statistical system. A consequence of the symposium was an agreement by the Committee on National Statistics to hold a workshop on the future of innovation in the federal statistical system. This workshop was held on June 29, 2010. The original statement of task for the workshop focused on three challenges to the statistical system: (1) the obstacles to innovative, focused research and development initiatives that could make statistical programs more cost effective; (2) a gap between emerging data visualization and communications technologies and the ability of statistical agencies to understand and capitalize on these developments for their data dissemination programs; and (3) the maturation of the information technology (IT) discipline and the difficulties confronting individual agencies in keeping current with best practice in IT regarding data confidentiality. This report, Facilitating Innovation in the Federal Statistical System, is a descriptive summary of what transpired at the workshop. It is therefore limited to the views and opinions of the workshop participants. However, it does not strictly follow the agenda of the workshop, which had four sessions. Instead, it is organized around the themes of the discussions, which migrated across the four sessions.

Introduction.Big data for twenty-first-century economic statistics: the future is now /Katharine G. Abraham, Ron S. Jarmin, Brian C. Moyer, and Matthew D. Shapiro --Toward comprehensive use of big data in economic statistics.Reengineering key national economic indicators /Gabriel Ehrlich, John Haltiwanger, Ron S. Jarmin, David Johnson, and Matthew D. Shapiro ;Big data in the US consumer price index: experiences and plans /Crystal G. Konny, Brendan K. Williams, and David M. Friedman ;Improving retail trade data products using alternative data sources /Rebecca J. Hutchinson ;From transaction data to economic statistics: constructing real-time, high-frequency, geographic measures of consumer spending /Aditya Aladangady, Shifrah Aron-Dine, Wendy Dunn, Laura Feiveson, Paul Lengermann, and Claudia Sahm ;Improving the accuracy of economic measurement with multiple data sources: the case of payroll employment data /Tomaz Cajner, Leland D. Crane, Ryan A. Decker, Adrian Hamins-Puertolas, and Christopher Kurz --Uses of big data for classification.Transforming naturally occurring text data into economic statistics: the case of online job vacancy postings /Arthur Turrell, Bradley Speigner, Jyldyz Djumalieva, David Copple, and James Thurgood ;Automating response evaluation for franchising questions on the 2017 economic census /Joseph Staudt, Yifang Wei, Lisa Singh, Shawn Klimek, J. Bradford Jensen, and Andrew Baer ;Using public data to generate industrial classification codes /John Cuffe, Sudip Bhattacharjee, Ugochukwu Etudo, Justin C. Smith, Nevada Basdeo, Nathaniel Burbank, and Shawn R. Roberts --Uses of big data for sectoral measurement.Nowcasting the local economy: using Yelp data to measure economic activity /Edward L. Glaeser, Hyunjin Kim, and Michael Luca ;Unit values for import and export price indexes: a proof of concept /Don A. Fast and Susan E. Fleck ;Quantifying productivity growth in the delivery of important episodes of care within the Medicare program using insurance claims and administrative data /John A. Romley, Abe Dunn, Dana Goldman, and Neeraj Sood ;Valuing housing services in the era of big data: a user cost approach leveraging Zillow microdata /Marina Gindelsky, Jeremy G. Moulton, and Scott A. Wentland --Methodological challenges and advances.Off to the races: a comparison of machine learning and alternative data for predicting economic indicators /Jeffrey C. Chen, Abe Dunn, Kyle Hood, Alexander Driessen, and Andrea Batch ;A machine learning analysis of seasonal and cyclical sales in weekly scanner data /Rishab Guha and Serena Ng ;Estimating the benefits of new products /W. Erwin Diewert and Robert C. Feenstra.

Governments have done much to leverage information technology to deploy e-government services, but much work remains before the vision of e-government can be fully realized. Information Technology Research, Innovation, and E-government examines the emerging visions for e-government, the technologies required to implement them, and approaches that can be taken to accelerate innovation and the transition of innovative information technologies from the laboratory to operational government systems. In many cases, government can follow the private sector in designing and implementing IT-based services. But there are a number of areas where government requirements differ from those in the commercial world, and in these areas government will need to act on its role as a "demand leader." Although researchers and government agencies may appear to by unlikely allies in this endeavor, both groups have a shared interest in innovation and meeting future needs. E-government innovation will require addressing a broad array of issues, including organization and policy as well as engineering practice and technology research and development, and each of these issues is considered in the book.

A wake-up call for America to create a new framework for democratizing data. Public data are foundational to our democratic system. People need consistently high-quality information from trustworthy sources. In the new economy, wealth is generated by access to data; government's job is to democratize the data playing field. Yet data produced by the American government are getting worse and costing more. In Democratizing Our Data, Julia Lane argues that good data are essential for democracy. Her book is a wake-up call to America to fix its broken public data system.