Outside of randomized experiments, association does not imply causation, and yet there is nothing defective about our knowledge that smoking causes lung cancer, a conclusion reached in the absence of randomized experimentation with humans. How is that possible? If observed associations do not identify causal effects in observational studies, how can a sequence of such associations become decisive? Two or more associations may each be susceptible to unmeasured biases, yet not susceptible to the same biases. An observational study has two evidence factors if it provides two comparisons susceptible to different biases that may be combined as if from independent studies of different data by different investigators, despite using the same data twice. If the two factors concur, then they may exhibit greater insensitivity to unmeasured biases than either factor exhibits on its own. Replication and Evidence Factors in Observational Studies includes four parts: A concise introduction to causal inference, making the book self-contained Practical examples of evidence factors from the health and social sciences with analyses in R The theory of evidence factors Study design with evidence factors A companion R package evident is available from CRAN.

Outside of randomized experiments, association does not imply causation, and yet there is nothing defective about our knowledge that smoking causes lung cancer, a conclusion reached in the absence of randomized experimentation with humans. How is that possible? If observed associations do not identify causal effects in observational studies, how can a sequence of such associations become decisive? Two or more associations may each be susceptible to unmeasured biases, yet not susceptible to the same biases. An observational study has two evidence factors if it provides two comparisons susceptible to different biases that may be combined as if from independent studies of different data by different investigators, despite using the same data twice. If the two factors concur, then they may exhibit greater insensitivity to unmeasured biases than either factor exhibits on its own. Replication and Evidence Factors in Observational studies has four parts: A concise introduction to causal inference, making the book self-contained. Practical examples of evidence factors from the health and social sciences with analyses in R. The theory of evidence factors. Study design with evidence factors. A companion R package evident is available from CRAN.

This thesis includes five chapters on evidence factors analysis of causal effect in various observational study settings. Each of these chapters can be read independently without knowledge of the content of any of the other chapters. Evidence factors allow for two independent analyses to be constructed from the same data set. When combining the evidence factors, the type-I error rate must be controlled to obtain valid inference. A powerful method is developed for controlling the familywise error rate for sensitivity analyses to unmeasured confounding with evidence factors. It is shown that the Bahadur efficiency of sensitivity analysis for the combined evidence is greater than for either evidence factor alone. The popular strategy of matching, for controlling the observed covariates, before inferring about the treatment effect, requires solving an optimization problem. This problem can be solved in polynomial time. In an evidence factors analysis we must consider multiple comparisons, thus the matching problem is often of matching at least three groups. This slightly different problem is much more difficult to solve. The third chapter proposes an approximation algorithm to solve this (and more practical versions of this) problem. We prove that the proposed algorithm provides a solution fast, that is provably not a lot further than the optimal solution that is difficult calculate. Two chapters that follow show the applicability of evidence factors analysis in more complicated study designs. The first of these two chapters considers a case-control study with multiple case definitions and the latter one considers studies with instrumental variables, where the instrument(s) may become invalid. The final chapter of the thesis develops a frequentist method for quantification of the degree of corroboration of causal hypothesis using the tool of evidence factors.

One of the pathways by which the scientific community confirms the validity of a new scientific discovery is by repeating the research that produced it. When a scientific effort fails to independently confirm the computations or results of a previous study, some fear that it may be a symptom of a lack of rigor in science, while others argue that such an observed inconsistency can be an important precursor to new discovery. Concerns about reproducibility and replicability have been expressed in both scientific and popular media. As these concerns came to light, Congress requested that the National Academies of Sciences, Engineering, and Medicine conduct a study to assess the extent of issues related to reproducibility and replicability and to offer recommendations for improving rigor and transparency in scientific research. Reproducibility and Replicability in Science defines reproducibility and replicability and examines the factors that may lead to non-reproducibility and non-replicability in research. Unlike the typical expectation of reproducibility between two computations, expectations about replicability are more nuanced, and in some cases a lack of replicability can aid the process of scientific discovery. This report provides recommendations to researchers, academic institutions, journals, and funders on steps they can take to improve reproducibility and replicability in science.

An observational study is an empiric investigation of effects caused by treatments when randomized experimentation is unethical or infeasible. Observational studies are common in most fields that study the effects of treatments on people, including medicine, economics, epidemiology, education, psychology, political science and sociology. The quality and strength of evidence provided by an observational study is determined largely by its design. Design of Observational Studies is both an introduction to statistical inference in observational studies and a detailed discussion of the principles that guide the design of observational studies. Design of Observational Studies is divided into four parts. Chapters 2, 3, and 5 of Part I cover concisely, in about one hundred pages, many of the ideas discussed in Rosenbaum’s Observational Studies (also published by Springer) but in a less technical fashion. Part II discusses the practical aspects of using propensity scores and other tools to create a matched comparison that balances many covariates. Part II includes a chapter on matching in R. In Part III, the concept of design sensitivity is used to appraise the relative ability of competing designs to distinguish treatment effects from biases due to unmeasured covariates. Part IV discusses planning the analysis of an observational study, with particular reference to Sir Ronald Fisher’s striking advice for observational studies, "make your theories elaborate." The second edition of his book, Observational Studies, was published by Springer in 2002.

This User’s Guide is a resource for investigators and stakeholders who develop and review observational comparative effectiveness research protocols. It explains how to (1) identify key considerations and best practices for research design; (2) build a protocol based on these standards and best practices; and (3) judge the adequacy and completeness of a protocol. Eleven chapters cover all aspects of research design, including: developing study objectives, defining and refining study questions, addressing the heterogeneity of treatment effect, characterizing exposure, selecting a comparator, defining and measuring outcomes, and identifying optimal data sources. Checklists of guidance and key considerations for protocols are provided at the end of each chapter. The User’s Guide was created by researchers affiliated with AHRQ’s Effective Health Care Program, particularly those who participated in AHRQ’s DEcIDE (Developing Evidence to Inform Decisions About Effectiveness) program. Chapters were subject to multiple internal and external independent reviews. More more information, please consult the Agency website: www.effectivehealthcare.ahrq.gov)

A wide-ranging discussion of factors that impede the cumulation of knowledge in the social sciences, including problems of transparency, replication, and reliability. Rather than focusing on individual studies or methods, this book examines how collective institutions and practices have (often unintended) impacts on the production of knowledge.