Questions about the reproducibility of scientific research have been raised in numerous settings and have gained visibility through several high-profile journal and popular press articles. Quantitative issues contributing to reproducibility challenges have been considered (including improper data measurement and analysis, inadequate statistical expertise, and incomplete data, among others), but there is no clear consensus on how best to approach or to minimize these problems. A lack of reproducibility of scientific results has created some distrust in scientific findings among the general public, scientists, funding agencies, and industries. While studies fail for a variety of reasons, many factors contribute to the lack of perfect reproducibility, including insufficient training in experimental design, misaligned incentives for publication and the implications for university tenure, intentional manipulation, poor data management and analysis, and inadequate instances of statistical inference. The workshop summarized in this report was designed not to address the social and experimental challenges but instead to focus on the latter issues of improper data management and analysis, inadequate statistical expertise, incomplete data, and difficulties applying sound statistic inference to the available data. Many efforts have emerged over recent years to draw attention to and improve reproducibility of scientific work. This report uniquely focuses on the statistical perspective of three issues: the extent of reproducibility, the causes of reproducibility failures, and the potential remedies for these failures.

Students and researchers in the health sciences are faced with greater opportunity and challenge than ever before. The opportunity stems from the explosion in publicly available data that simultaneously informs and inspires new avenues of investigation. The challenge is that the analytic tools required go far beyond the standard methods and models of basic statistics. This textbook aims to equip health care researchers with the most important elements of a modern health analytics toolkit, drawing from the fields of statistics, health econometrics, and data science. This textbook is designed to overcome students’ anxiety about data and statistics and to help them to become confident users of appropriate analytic methods for health care research studies. Methods are presented organically, with new material building naturally on what has come before. Each technique is motivated by a topical research question, explained in non-technical terms, and accompanied by engaging explanations and examples. In this way, the authors cultivate a deep (“organic”) understanding of a range of analytic techniques, their assumptions and data requirements, and their advantages and limitations. They illustrate all lessons via analyses of real data from a variety of publicly available databases, addressing relevant research questions and comparing findings to those of published studies. Ultimately, this textbook is designed to cultivate health services researchers that are thoughtful and well informed about health data science, rather than data analysts. This textbook differs from the competition in its unique blend of methods and its determination to ensure that readers gain an understanding of how, when, and why to apply them. It provides the public health researcher with a way to think analytically about scientific questions, and it offers well-founded guidance for pairing data with methods for valid analysis. Readers should feel emboldened to tackle analysis of real public datasets using traditional statistical models, health econometrics methods, and even predictive algorithms. Accompanying code and data sets are provided in an author site: https://roman-gulati.github.io/statistics-for-health-data-science/

One of the pathways by which the scientific community confirms the validity of a new scientific discovery is by repeating the research that produced it. When a scientific effort fails to independently confirm the computations or results of a previous study, some fear that it may be a symptom of a lack of rigor in science, while others argue that such an observed inconsistency can be an important precursor to new discovery. Concerns about reproducibility and replicability have been expressed in both scientific and popular media. As these concerns came to light, Congress requested that the National Academies of Sciences, Engineering, and Medicine conduct a study to assess the extent of issues related to reproducibility and replicability and to offer recommendations for improving rigor and transparency in scientific research. Reproducibility and Replicability in Science defines reproducibility and replicability and examines the factors that may lead to non-reproducibility and non-replicability in research. Unlike the typical expectation of reproducibility between two computations, expectations about replicability are more nuanced, and in some cases a lack of replicability can aid the process of scientific discovery. This report provides recommendations to researchers, academic institutions, journals, and funders on steps they can take to improve reproducibility and replicability in science.

The Office of the Under Secretary of Defense (Personnel & Readiness), referred to throughout this report as P&R, is responsible for the total force management of all Department of Defense (DoD) components including the recruitment, readiness, and retention of personnel. Its work and policies are supported by a number of organizations both within DoD, including the Defense Manpower Data Center (DMDC), and externally, including the federally funded research and development centers (FFRDCs) that work for DoD. P&R must be able to answer questions for the Secretary of Defense such as how to recruit people with an aptitude for and interest in various specialties and along particular career tracks and how to assess on an ongoing basis service members' career satisfaction and their ability to meet new challenges. P&R must also address larger-scale questions, such as how the current realignment of forces to the Asia-Pacific area and other regions will affect recruitment, readiness, and retention. While DoD makes use of large-scale data and mathematical analysis in intelligence, surveillance, reconnaissance, and elsewhereâ€"exploiting techniques such as complex network analysis, machine learning, streaming social media analysis, and anomaly detectionâ€"these skills and capabilities have not been applied as well to the personnel and readiness enterprise. Strengthening Data Science Methods for Department of Defense Personnel and Readiness Missions offers and roadmap and implementation plan for the integration of data analysis in support of decisions within the purview of P&R.

The primary function of the intelligence analyst is to make sense of information about the world, but the way analysts do that work will look profoundly different a decade from now. Technological changes will bring both new advances in conducting analysis and new risks related to technologically based activities and communications around the world. Because these changes are virtually inevitable, the Intelligence Community will need to make sustained collaboration with researchers in the social and behavioral sciences (SBS) a key priority if it is to adapt to these changes in the most productive ways. A Decadal Survey Of The Social and Behavioral Sciences provides guidance for a 10-year research agenda. This report identifies key opportunities in SBS research for strengthening intelligence analysis and offers ideas for integrating the knowledge and perspectives of researchers from these fields into the planning and design of efforts to support intelligence analysis.

The environment for obtaining information and providing statistical data for policy makers and the public has changed significantly in the past decade, raising questions about the fundamental survey paradigm that underlies federal statistics. New data sources provide opportunities to develop a new paradigm that can improve timeliness, geographic or subpopulation detail, and statistical efficiency. It also has the potential to reduce the costs of producing federal statistics. The panel's first report described federal statistical agencies' current paradigm, which relies heavily on sample surveys for producing national statistics, and challenges agencies are facing; the legal frameworks and mechanisms for protecting the privacy and confidentiality of statistical data and for providing researchers access to data, and challenges to those frameworks and mechanisms; and statistical agencies access to alternative sources of data. The panel recommended a new approach for federal statistical programs that would combine diverse data sources from government and private sector sources and the creation of a new entity that would provide the foundational elements needed for this new approach, including legal authority to access data and protect privacy. This second of the panel's two reports builds on the analysis, conclusions, and recommendations in the first one. This report assesses alternative methods for implementing a new approach that would combine diverse data sources from government and private sector sources, including describing statistical models for combining data from multiple sources; examining statistical and computer science approaches that foster privacy protections; evaluating frameworks for assessing the quality and utility of alternative data sources; and various models for implementing the recommended new entity. Together, the two reports offer ideas and recommendations to help federal statistical agencies examine and evaluate data from alternative sources and then combine them as appropriate to provide the country with more timely, actionable, and useful information for policy makers, businesses, and individuals.

In the Third Edition of The Psychology Research Handbook editors Frederick T. L. Leong and James T. Austin have assembled experienced expert researchers to provide graduate students and research assistants with a comprehensive framework for conducting many types of psychology research. The book is organized around the idea of a "research script," following the step-by-step process of research planning, design, data collection, analysis, and disseminating research. Many chapters are coauthored by advanced graduate students to give their fellow students a sense of real-world research, adding to the clarity and practicality of many chapters. Students and instructors alike will appreciate chapters on topics typically missing from introductory methods texts, including applying for research grants, dealing with journal editors and reviewers, working within research teams, and conducting cross-cultural research. Structures such as recommended readings and exercises guide students to develop and expand their research skills. New chapters include Power and Evidence, IRB as Critical Collaborators in Research, Alternative Data Collection Strategies, Structural Equation Modeling and Replicability and Reproducibility. A comprehensive, easy-to-understand guide to the entire research process, this book quickly and efficiently equips advanced students and research assistants to conduct a full research project.