This User’s Guide is a resource for investigators and stakeholders who develop and review observational comparative effectiveness research protocols. It explains how to (1) identify key considerations and best practices for research design; (2) build a protocol based on these standards and best practices; and (3) judge the adequacy and completeness of a protocol. Eleven chapters cover all aspects of research design, including: developing study objectives, defining and refining study questions, addressing the heterogeneity of treatment effect, characterizing exposure, selecting a comparator, defining and measuring outcomes, and identifying optimal data sources. Checklists of guidance and key considerations for protocols are provided at the end of each chapter. The User’s Guide was created by researchers affiliated with AHRQ’s Effective Health Care Program, particularly those who participated in AHRQ’s DEcIDE (Developing Evidence to Inform Decisions About Effectiveness) program. Chapters were subject to multiple internal and external independent reviews. More more information, please consult the Agency website: www.effectivehealthcare.ahrq.gov)

Randomized studies are designed to estimate the average treatment effect (ATE) of an intervention. Individuals may derive quantitatively, or even qualitatively, different effects from the ATE, which is called the heterogeneity of treatment effect. It is important to detect the existence of heterogeneity in the treatment responses, and identify the different sub-populations. Two corresponding statistical methods will be discussed in this talk: a hypothesis testing procedure and a mixture-model based approach. The hypothesis testing procedure was constructed to test for the existence of a treatment effect in sub-populations. The test is nonparametric, and can be applied to all types of outcome measures. A key innovation of this test is to build stochastic search into the test statistic to detect signals that may not be linearly related to the multiple covariates. Simulations were performed to compare the proposed test with existing methods. Power calculation strategy was also developed for the proposed test at the design stage. The mixture-model based approach was developed to identify and study the sub-populations with different treatment effects from an intervention. A latent binary variable was used to indicate whether or not a subject was in a sub-population with average treatment benefit. The mixture-model combines a logistic formulation of the latent variable with proportional hazards models. The parameters in the mixture-model were estimated by the EM algorithm. The properties of the estimators were then studied by the simulations. Finally, all above methods were applied to a real randomized study in a low ejection fraction population that compared the Implantable Cardioverter Defibrillator (ICD) with conventional medical therapy in reducing total mortality.

The Agency for Healthcare Research and Quality (AHRQ) commissioned the RTI International–University of North Carolina at Chapel Hill (RTI-UNC) Evidence-based Practice Center (EPC) to explore how systematic review groups have dealt with clinical heterogeneity and to seek out best practices for addressing clinical heterogeneity in systematic reviews (SRs) and comparative effectiveness reviews (CERs). Such best practices, to the extent they exist, may enable AHRQ's EPCs to address critiques from patients, clinicians, policymakers, and other proponents of health care about the extent to which “average” estimates of the benefits and harms of health care interventions apply to individual patients or to small groups of patients sharing similar characteristics. Such users of reviews often assert that EPC reviews typically focus on broad populations and, as a result, often lack information relevant to patient subgroups that are of particular concern to them. More important, even when EPCs evaluate literature on homogeneous groups, there may be varying individual treatment for no apparent reason, indicating that average treatment effect does not point to the best treatment for any given individual. Thus, the health care community is looking for better ways to develop information that may foster better medical care at a “personal” or “individual” level. To address our charge for this methods project, the EPC set out to answer six key questions (KQ). Key questions for methods report on clinical heterogeneity include: 1. What is clinical heterogeneity? a. How has it been defined by various groups? b. How is it distinct from statistical heterogeneity? c. How does it fit with other issues that have been addressed by the AHRQ Methods Manual for CERs? 2. How have systematic reviews dealt with clinical heterogeneity in the key questions? a. What questions have been asked? b. How have they pre-identified population subgroups with common clinical characteristics that modify their intervention-outcome association? c. What are best practices in key questions and how these subgroups have been identified? 3. How have systematic reviews dealt with clinical heterogeneity in the review process? a. What do guidance documents of various systematic review groups recommend? b. How have EPCs handled clinical heterogeneity in their reviews? c. What are best practices in searching for and interpreting results for particular subgroups with common clinical characteristics that may modify their intervention-outcome association? 4. What are critiques in how systematic reviews handle clinical heterogeneity? a. What are critiques from specific reviews (peer and public) on how EPCs handled clinical heterogeneity? b. What general critiques (in the literature) have been made against how systematic reviews handle clinical heterogeneity? 5. What evidence is there to support how to best address clinical heterogeneity in a systematic review? 6. What questions should an EPC work group on clinical heterogeneity address? Heterogeneity (of any type) in EPC reviews is important because its appearance suggests that included studies differed on one or more dimensions such as patient demographics, study designs, coexisting conditions, or other factors. EPCs then need to clarify for clinical and other audiences, collectively referred to as stakeholders, what are the potential causes of the heterogeneity in their results. This will allow the stakeholders to understand whether and to what degree they can apply this information to their own patients or constituents. Of greatest importance for this project was clinical heterogeneity, which we define as the variation in study population characteristics, coexisting conditions, cointerventions, and outcomes evaluated across studies included in an SR or CER that may influence or modify the magnitude of the intervention measure of effect (e.g., odds ratio, risk ratio, risk difference).

There is a growing interest in estimating heterogeneous treatment effects in randomized and observational studies. However, most of the work relies on the assumption of ignorability, or no unmeasured confounding on the treatment effect. While instrumental variables (IV) are a popular technique to control for unmeasured confounding, there has been little research conducted to study heterogeneous treatment effects with the use of an IV. This dissertation introduces methods using an IV to discover novel subgroups, estimate their heterogeneous treatment effects, and identify individualized treatment rules (ITR) when ignorability is expected to be violated. In Chapter 2, we present a two-part algorithm to estimate heterogeneous treatment effects and detect novel subgroups using an IV with matching. The first part uses interpretable machine learning techniques, such as classification and regression trees, to discover potential effect modifiers. The second part uses closed testing to test for statistical significance of each effect modifier while strongly controlling the familywise error rate. We apply this method on the Oregon Health Insurance Experiment, estimating the effect of Medicaid on the number of days an individual's health does not impede their usual activities by using a randomized lottery as an instrument. In Chapter 3, we generalize methods to identify ITR using a binary IV to using multiple, discrete valued instruments, or equivalently, multilevel instruments. Several new problems arise when generalizing to multilevel instruments, requiring novel solutions. In particular, multilevel IV give rise to many latent subgroups that may experience heterogeneous treatment effects. Additionally, it may be unclear how to combine and compare the different levels of the IV to estimate treatment heterogeneity. We provide methods that use a prediction of the latent subgroup to identify optimal ITR, and methods to dynamically combine levels of the multilevel IV to estimate the heterogeneous treatment effects, effectively individualizing estimation of an ITR. Further, we provide and discuss necessary and sufficient conditions to identify an optimal ITR using a multilevel IV. We apply our methods to identify an ITR for two competing treatments, carotid endarterectomy and carotid artery stenting, on preventing stroke or death within 30 days of their index procedure.

Healthcare providers, consumers, researchers and policy makers are inundated with unmanageable amounts of information, including evidence from healthcare research. It has become impossible for all to have the time and resources to find, appraise and interpret this evidence and incorporate it into healthcare decisions. Cochrane Reviews respond to this challenge by identifying, appraising and synthesizing research-based evidence and presenting it in a standardized format, published in The Cochrane Library (www.thecochranelibrary.com). The Cochrane Handbook for Systematic Reviews of Interventions contains methodological guidance for the preparation and maintenance of Cochrane intervention reviews. Written in a clear and accessible format, it is the essential manual for all those preparing, maintaining and reading Cochrane reviews. Many of the principles and methods described here are appropriate for systematic reviews applied to other types of research and to systematic reviews of interventions undertaken by others. It is hoped therefore that this book will be invaluable to all those who want to understand the role of systematic reviews, critically appraise published reviews or perform reviews themselves.

Doing Meta-Analysis with R: A Hands-On Guide serves as an accessible introduction on how meta-analyses can be conducted in R. Essential steps for meta-analysis are covered, including calculation and pooling of outcome measures, forest plots, heterogeneity diagnostics, subgroup analyses, meta-regression, methods to control for publication bias, risk of bias assessments and plotting tools. Advanced but highly relevant topics such as network meta-analysis, multi-three-level meta-analyses, Bayesian meta-analysis approaches and SEM meta-analysis are also covered. A companion R package, dmetar, is introduced at the beginning of the guide. It contains data sets and several helper functions for the meta and metafor package used in the guide. The programming and statistical background covered in the book are kept at a non-expert level, making the book widely accessible. Features • Contains two introductory chapters on how to set up an R environment and do basic imports/manipulations of meta-analysis data, including exercises • Describes statistical concepts clearly and concisely before applying them in R • Includes step-by-step guidance through the coding required to perform meta-analyses, and a companion R package for the book

Clinical trials are used to elucidate the most appropriate preventive, diagnostic, or treatment options for individuals with a given medical condition. Perhaps the most essential feature of a clinical trial is that it aims to use results based on a limited sample of research participants to see if the intervention is safe and effective or if it is comparable to a comparison treatment. Sample size is a crucial component of any clinical trial. A trial with a small number of research participants is more prone to variability and carries a considerable risk of failing to demonstrate the effectiveness of a given intervention when one really is present. This may occur in phase I (safety and pharmacologic profiles), II (pilot efficacy evaluation), and III (extensive assessment of safety and efficacy) trials. Although phase I and II studies may have smaller sample sizes, they usually have adequate statistical power, which is the committee's definition of a "large" trial. Sometimes a trial with eight participants may have adequate statistical power, statistical power being the probability of rejecting the null hypothesis when the hypothesis is false. Small Clinical Trials assesses the current methodologies and the appropriate situations for the conduct of clinical trials with small sample sizes. This report assesses the published literature on various strategies such as (1) meta-analysis to combine disparate information from several studies including Bayesian techniques as in the confidence profile method and (2) other alternatives such as assessing therapeutic results in a single treated population (e.g., astronauts) by sequentially measuring whether the intervention is falling above or below a preestablished probability outcome range and meeting predesigned specifications as opposed to incremental improvement.