Commentary| Volume 11, ISSUE 10, P707-711, October 2009

Comparative effectiveness research and genomic medicine: An evolving partnership for 21st century medicine


      The American Recovery and Reinvestment Act has provided resources for comparative effectiveness research that will lead to evidence-based decisions about health and health care choices. Some have voiced concerns that evidence-based comparative effectiveness research principles are only relevant to “average” patients and not as much to individuals with unique combinations of genes, exposures and disease outcomes, intrinsic to genomic medicine. In this commentary, we argue that comparative effectiveness research and genomic medicine not only can and should coexist but also they will increasingly benefit from each other. The promise and success of genomic medicine will depend on rigorous comparative effectiveness research to compare outcomes for genome-based applications in practice to traditional non–genome-based approaches. In addition, the success of comparative effectiveness research will depend on developing new methods and clinical research infrastructures to integrate genome-based personalized perspectives into point of care decisions by patients and providers. There is a need to heal the apparent schism between genomic medicine and comparative effectiveness research to enhance knowledge-driven practice of medicine in the 21st century.


      What is comparative effectiveness research?

      CER has been defined in many ways.
      • Tuckson R.V.
      Challenges and opportunities for evidence-based genetics practice.
      • Rich E.C.
      The policy debate over public investment in comparative effectiveness research.

      Congressional Budget Office: Research on the comparative effectiveness of medical treatments. Available at: Accessed April 8, 2009.

      111th Congress of the United States Washington DC. Conference report to accompany HR 1. 2009; 111–116: 1, 1–780.

      The Congressional Budget Office defines CER as “a rigorous evaluation of the impact of different options that are available for treating a given medical condition for a particular set of patients.”
      • Rich E.C.
      The policy debate over public investment in comparative effectiveness research.
      These options include tests, devices, drugs, biologics, counseling, and surgical procedures. The House Senate Conference Report went on to emphasize the personalized nature of such information “… the conferees recognize that a “one-size-fits-all” approach to patient treatment is not the most medically appropriate solution to treating various conditions and include language to ensure that subpopulations are considered when research is conducted or supported.”

      Congressional Budget Office: Research on the comparative effectiveness of medical treatments. Available at: Accessed April 8, 2009.

      The Department of Health and Human Services recently published the definition of CER established by the Federal Coordinating Council for CER, which focuses on the role of CER in informing physicians, patients, and other decision-makers.

      Department of Health and Human Services. Available at: Accessed May 28 2009.

      The purpose of this research is to inform patients, providers, and decision-makers, responding to their expressed needs, about which interventions are most effective for which patients under specific circumstances and for diverse patient populations.
      As research intended to inform clinicians and individuals making real-world decisions, CER can use all the methodological approaches available in the clinical and population research armamentarium. These include various types of observational studies, randomized clinical trials (RCTs), and systematic reviews of evidence and modeling. Most RCTs are focused on efficacy—the extent to which an intervention produces a beneficial result under ideal conditions, and very few RCTs are focused on effectiveness—the extent to which a specific intervention, when used under ordinary circumstances, does what it is intended to do. These types of effectiveness trials are sometimes referred to as “large simple,” “pragmatic,” or “practical” trials.
      • Tunis S.R.
      • Stryer D.B.
      • Clancy C.M.
      Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy.
      Pragmatic trials compare clinically relevant alternatives, they enroll a diverse study population, they recruit from a variety of practice settings, and they measure a broad range of health outcomes. In our mind, there is a need to conduct CER studies to cover the continuum from efficacy to effectiveness in GM.

      How will genomic medicine benefit from comparative effectiveness research?

      With accelerating discoveries about the human genome, we are faced with mounting expectations of a new era of personalized health care and disease prevention based on genomic tools and technologies.

      Zerhouni E. NIH Press release: NIH seeks input on proposed repository for genetic information. Available at: Accessed September 1, 2006.

      • Burke W.
      • Psaty B.
      Personalized medicine in the era of genomics.
      Although major progress continues in our understanding of the biology of disease and development of new technologies, unfortunately, there are only a few genomic applications that are ready for use in clinical practice.
      • Khoury M.J.
      • Berg A.
      • Coates R.
      • Evans J.
      • Teutsch S.M.
      • Bradley L.A.
      The evidence dilemma in genomic medicine.
      For therapeutic-based genome applications, randomized controlled clinical trials are usually needed to show efficacy and effectiveness. For diagnostic-based genomic applications, evidence is needed in three domains
      • Haddow J.E.
      • Palomaki G.E.
      ACCE: a model process for evaluating emerging data on genetic tests.
      : analytic validity (how well tests perform in the laboratory), clinical validity (how well do tests correlate with clinical endpoints), and clinical utility (whether the use of testing improves health outcomes). These three domains also apply to pharmacogenomic testing and for genomic tests that are used for prediction and prognosis. For example, most discovered genetic variants are poor predictors for future disease and have thus poor clinical validity.
      • Kraft P.
      • Hunter D.J.
      Genetic risk prediction—are we there yet?.
      Even with strongly established genetic associations such as Factor V Leiden and recurrent venous thromboembolism, it is not clear whether testing for Factor V Leiden can improve clinical outcomes.
      • Segal J.B.
      • Brotman D.J.
      • Necochea A.J.
      Predictive value of factor V Leiden and prothrombin G20210A in adults with venous thromboembolism and in family members of those with a mutation: a systematic review.
      One genomic application with documented clinical utility is somatic HER2 tumor testing to target trastuzumab treatment for patients with breast cancer.
      • Phillips K.A.
      Closing the evidence gap in the use of emerging testing technologies in clinical practice.
      Even for this example, there are gaps in evidence in implementation. Because of the high cost of trastuzumab therapy (approximately $100,000 annually), there is an urgent need for evidence on how to most efficiently target such therapy. Although there is no “gold standard” method to determine HER2 status in tumor tissue, fluorescence in situ hybridization is widely assumed to be a better predictor of treatment response, whereas immunohistochemistry costs less and is easier to perform in many laboratories. Problems with test accuracy were acknowledged by the company.
      • Kraft P.
      • Hunter D.J.
      Genetic risk prediction—are we there yet?.
      Although a professional panel has recently reached consensus on the approach to HER2 testing,
      • Wolff A.C.
      • Hammond M.E.
      • Schwartz J.N.
      American Society of Clinical Oncology/College of American Pathologists guideline recommendations for human epidermal growth factor receptor 2 testing in breast cancer.
      an Agency for Health care Quality (AHRQ)-sponsored report identified gaps in evidence on outcomes of addition of trastuzumab to chemotherapy in patients with HER2 equivocal, discordant, or negative.

      Agency for Healthcare Research Quality. HER2 testing to manage patients with breast or other solid tumors. 2008; Available at: Accessed April 15, 2009.

      Most genomic applications to date are further from the bedside than HER2 with no or little documentation of clinical utility. An example is genetic testing to inform anticoagulation therapy with warfarin. The CYP2C9 and VKORC1 genes are implicated in warfarin and vitamin K metabolism, and variants in these genes are consistently associated with warfarin bleeding complications. However, a recent small RCT of pharmacogenetic-guided dosing did not show a statistically significant difference in its primary endpoint—percent out-of-range international normalized ratios (INRs), which is a surrogate outcome.
      • Couma-Gen Investigators
      Randomized trial of genotype-guided versus standard warfarin dosing in patients initiating oral anticoagulation.
      We know that many factors such as drug–drug and diet–drug interactions, adherence to medication, and regular INR testing and dosing changes based on test results have an impact on outcomes of warfarin therapy. The central question is the incremental health benefit of using genetic factors to determine the initial dose of warfarin over and above the well-established factors that can be used to determine dosage. A recent systematic review did not find sufficient evidence to support the use of pharmacogenetics to guide warfarin therapy.
      • Kangelaris K.N.
      • Bent S.
      • Nussbaum R.L.
      • Garcia D.A.
      • Tice J.A.
      Genetic testing before anticoagulation? A systematic review of pharmacogenomic dosing of warfarin.
      Additional clinical trials are needed to define whether using genetic testing can provide added value to monitoring INR in improving outcomes of warfarin therapy.
      • Lackner T.A.
      Pharmacogenomic dosing of warfarin: ready or not?.
      Another example of unknown clinical utility of genomic information is the 9p21 genetic variant associated with coronary heart disease, which has been postulated as a tool for future prediction of heart disease and target cholesterol but currently lacks direct evidence of clinical utility.
      • Ioannidis J.P.
      Personalized genetic prediction: too limited, too expensive or too soon.
      Finally, although there are numerous examples of genetic susceptibility to cigarette smoking and adverse health effects, it is not clear what would be the added value of targeted cigarette smoking cessation efforts based on individual genetic susceptibility compared with the same interventions without genetic targeting.
      • Carlsten C.
      • Burke W.
      Potential for genetics to promote public health: genetics research on smoking suggests caution about expectations.
      The premature introduction of new genome-based technologies into health care settings could distract clinicians and patients from focusing attention on use of interventions of proven benefit. In the face of inconclusive evidence, coverage and reimbursement policies by public and private payers will be idiosyncratic, contributing to variations in access to technologies of both proven and unknown benefits.

      Zerhouni E. NIH Press release: NIH seeks input on proposed repository for genetic information. Available at: Accessed September 1, 2006.

      To address evidentiary needs for GM, the Centers for Disease Control and Prevention launched in 2004, the Evaluation of Genomic Applications in Practice and Prevention (EGAPP) Initiative, which supports the development and implementation of a rigorous, evidence-based process for evaluating genetic tests and other genomic applications for clinical and public health practice in the United States.

      EGAPP—Evaluation of Genomic Applications in Practice and Prevention website. Available at: Accessed February 25, 2009.

      An independent EGAPP Working Group selects topics, oversees the systematic review of evidence by the AHRQ evidence-based Practice Centers and in-house reviews, and makes recommendations based on that evidence. EGAPP has published its methods for the evaluation of genomic applications in practice,

      The Community Guide. Available at: Accessed April 8 2009.

      which were adapted from existing methods in evaluating interventions from professional organizations and advisory committees, task forces (e.g., US Preventive Services Task Force [USPSTF,

      Agency for HealthCare Research Quality: US Preventive Services Task Force (USPTSF). Available at: Accessed April 8, 2009.

      ], and Task Force on Community Preventive Services.

      The Community Guide. Available at: Accessed April 8 2009.

      By January 2009, five evidence reports and four evidence recommendations were published in this journal.
      • EGAPP
      working group-recommendation statements.
      Based on the evidence reports and clinical and social contextual issues, the working group develops recommendation statements summarizing the current knowledge about the clinical validity and utility of the genetic test, provides guidance on appropriate use of the test, and defines key knowledge gaps and needed research. Four of the five topics examined, to date, have returned major knowledge gaps in clinical validity and utility with insufficient evidence for routine use in practice.
      • EGAPP
      working group-recommendation statements.
      Because many genomic tests are being developed for disease prediction and prevention, they should meet the same evidentiary standards as other screening tests. Such screening tests expose large numbers of healthy people to potential harms from false-positive results (such as anxiety and “labeling,” as well as additional invasive testing and treatment) or from false-negative results (such as false reassurance and attendant lapses in personalized risk factor reduction). As a result, groups formulating evidence-based clinical recommendations such as EGAPP and USPSTF have required at least a moderate level of certainty of the benefits of screening outweighing the harms.

      Agency for HealthCare Research Quality: US Preventive Services Task Force (USPTSF). Available at: Accessed April 8, 2009.

      • Teutsch S.M.
      • Bradley L.A.
      • Palomaki G.
      The Evaluation of Genomic Applications in Practice and Prevention (EGAPP) initiative: methods of the EGAPP working group.
      In Table 1, we show, in the context of CER, the EGAPP approach to the evaluation of clinical validity and clinical utility of genomic information by the different types of intended use (ranging from diagnostic tests to tests predicting drug response and adverse reactions). For all current intended uses, CER can be an important tool to assess both the clinical validity and utility of these applications, using the examples above and others mentioned in table. These examples range from single-gene disorders to common multifactorial conditions where complex genetic information can be used, as well as drug-related interventions (pharmacogenomics). Because most genomic applications will be competing with current clinical practice using other tests, evidence is needed to show whether genomic tests provide clinically meaningful incremental benefits in real-world settings.
      Table 1Categories of gene-based test applications and the role of comparative effectiveness research in evaluating their clinical validity and utility
      Table thumbnail fx1

      How will comparative effectiveness research benefit from genomic medicine?

      The unrelenting progress in genome-based discoveries for many diseases for both diagnostics and therapeutics will provide an important impetus for enhancing and even shaping CER research questions, methods, and capacity for years to come. Clinicians have long recognized that the skilled practice of evidence-based medicine must incorporate an understanding of the individual patient's unique characteristics and circumstances, and recent legislation has confirmed the importance of this personal patient perspective in the CER. As the number of novel gene-based findings is likely to expand, it will become increasingly difficult for traditional clinical trials to incorporate all these potential “personalizing” factors, including genomics. CER methods must address (i) the sheer volume of gene-based applications and other diagnostic tests; (ii) the timeliness for collecting evidence; (iii) the costs of doing large-scale research in this new field; and (iv) standards for evidence for these new applications. GM will increasingly influence CER approaches to establish the appropriate threshold of certainty that an innovation is superior to usual care in various scenarios, as well as the research methods and analytic techniques needed to detect such effects. The factors involved in establishing decision thresholds likely will include anticipated effect sizes of interventions, nature of the risks and benefits, patient-specific factors (including genetics) as well as individual preferences for various outcomes. All of these considerations will in turn influence the appropriate type of CER required to provide individual patients with personalized answers to the clinical evidence questions of the future.

      The Brookings Institution: implementing comparative effectiveness research: priorities impact. Available at:∼/media/Files/events/2009/0609_health_care_cer/20090609_health_care_cer.pdf Accessed June 29, 2009.

      To enhance the ability of rapid data collection and analysis on GM, many hope to assemble large networks of clinical data to support “virtual clinical research” with infrastructure integrating clinical data “to enable patients to be molecularly profiled and pre-enrolled in clinical research” and participate in “on-demand clinical trials” and cohort studies.

      BIGHealth consortium. Available at: Accessed April 8 2009.

      Although these approaches can provide valuable information, there is still concern about adequate representation of different patient populations that may lead to a lack of generalizability. Methodological issues (such as selection, information bias, and confounding) still have to be addressed. Many envision a fusion of clinical and molecular data using health information technology platforms in large databases that can enhance effectiveness studies, overall and for defined subpopulations (based on genomic and molecular data). Woodcock
      • Woodcock J.
      The human genome and translational research: how much evidence is enough?.
      and Hudson
      • Hudson K.
      The health benefits of genomics: out with the old, in with the new.
      recently argued that the disproportionate emphasis on genomics discovery needs to be balanced by new translational approaches to the development and dissemination of evidence-based clinical guidelines. They note that RCTs may not be always necessary or feasible for evaluating diagnostic or predictive genetic tests, but both authors emphasize the need for a high standard of evidence when such tests inform high-stakes decisions. Accordingly, Califf and Ginsburg
      • Califf R.M.
      • Ginsburg G.S.
      Organizational improvements to enhance modern clinical epidemiology.
      argue that it is time for new models of rapid collection of evidence anchored by “multiple interdisciplinary investigative teams that develop ‘disease state models’ … by integrating fundamental knowledge with clinical and molecular databases and population records.” Innovations such as these will need validation themselves. For example, much more research will be required before we can be confident that our genomic and pathophysiological understanding of disease processes can provide a valid basis for estimating intervention effects in health outcomes for genetically defined subpopulations. Integrating GM into thinking about how to measure patient benefit will necessitate bringing together different academic disciplines to explore enhanced models for evidence-based medicine. Multidisciplinary teams that engage behavioral and social scientists, medical ethicists, policy experts, mathematicians, physical and biological scientists in addition to clinicians, pharmacists, health services researchers, economists, epidemiologists, and statisticians will function to develop multilevel and innovative systems approaches to build CER capability and tools in GM. This approach would require the implementation of health systems with interoperable electronic health records (EHRs) and biobanks, which could rapidly aggregate clinical and molecular data for cross-sectional and longitudinal information on interventions, risk factors, and clinical outcomes. Clearly, our current research infrastructure in linking genetic testing data with clinical and other outcomes is woefully inadequate as recently documented by a CDC-AHRQ-sponsored review.

      Agency for HealthCare Research Quality: infrastructure to monitor utilization and outcomes for gene-based technologies (2008). Available at: Accessed April 8, 2009.

      Data collection efforts in health care settings (e.g., cancer research network

      National Cancer Institute. Cancer Research Network. Available at: Accessed April 8 2009.

      ) are also being supplemented by large-scale biobanks in population studies that examine genetic, environmental, and other factors in relation to health outcomes.

      Public Population Project in Genomics. Available at: P3G consortium Accessed April 8 2009.

      We need to develop and use health information technology that allows faster information flow, so that evidence-based guidelines are used more quickly in practice, through clinical decision support tools.

      Agency for Healthcare Research Quality. Computer-based clinical decision support (CDS) tools for gene-based tests used in breast cancer. Available at: Accessed September 29, 2008.

      Many of the limitations in our current databases derive from the fact that they were created for administrative and billing purposes and not for research; hence, there is a paucity of detailed clinical patient and outcome data. However, our investments in health information technology and increasing use of EHRs will give us a unique opportunity to design databases that can be used to improve delivery of care as well as improve the conduct of practice-based clinical research, while protecting privacy and confidentiality of a person's information. The databases can be the traditional large, centralized, aggregated databases or the newly emerging distributed research networks. For example, AHRQ has funded two pilot projects on distributed research: one based on the ambulatory care setting that works across most EHR systems

      Agency for Healthcare Research Quality. Distributed Network for Ambulatory Research in Therapeutics (DARTNet). Available at: Accessed April 15, 2009.

      and the other is being tested in large health maintenance organizations.

      Agency for Healthcare Research Quality. Developing a Distributed Research Network to Conduct Population-based Studies and Safety Surveillance. Available at: Accessed April 15, 2009.

      An attractive feature of EHR-based databases is that they can collect information in near real time as well as new data at the point of care as needed. When combined with tools such as natural language processing, these databases can greatly enhance the richness of clinical treatment, risk factor, and health outcome information far beyond that available in traditional administrative claims databases. Recently, Kowawoto et al.
      • Kowawoto K.
      • Lobach D.F.
      • Willard H.F.
      • Ginsburg G.S.
      A national clinical decision support infrastructure to enable the widespread and consistent practice of genomic and personalized medicine.
      recommended the establishment of a national decision support infrastructure that assists clinicians in their use of genomic assays to guide disease prevention, diagnosis, and therapy. Components of this infrastructure would include standardized representation of patient data across health information systems, centrally managed repositories of medical knowledge, and standardized approaches for applying these knowledge resources to generate patient-specific care recommendations.
      Thus, the current and expanding interest in GM will help the overall CER enterprise by accelerating the creation of clinical and population research infrastructures that could be used for all facets of CER including GM. In addition to accelerating the clinical translation research infrastructure, GM could also help the development of new approaches toward knowledge synthesis from basic, clinical, and population sciences that would enhance current methods of evidence reviews that rely on empirical published findings. Improved methods to conduct modeling and economic evaluations will allow us to incorporate them as part of systematic evidence reviews and improve evidence-based decisions. This will facilitate more rapid and continuous assessments of the evolving evidence coming from multiple fields. For example, cancer control and prevention have been informed by the increasing use of modeling through the Cancer Intervention and Surveillance Modeling Network, a consortium of National Cancer Institute-sponsored investigators that use statistical modeling to improve our understanding of cancer control interventions in prevention, screening, and treatment on population trends in incidence and mortality.

      National Cancer Institute. Cancer Intervention and Surveillance Modeling Network (CISNET). Available at: Accessed April 8, 2009.

      Although modeling efforts to date have been used mainly to refine recommendations (e.g., age cutoffs, frequency of testing, or test sequences), they could potentially be adapted to address “what if” questions related to CER in GM and point to important knowledge gaps that require additional clinical and population data collection. The EGAPP working group and the USPSTF have also used modeling in their appraisals.
      In summary, GM and CER will evolve in a symbiotic and mutually beneficial manner. The generation of large and rapidly evolving information from GM will contribute to building the translational research and informatics infrastructures that are absolutely crucial for demonstrating the effectiveness or lack thereof of GM in health care and disease prevention. As part of American Recovery and Reinvestment Act funds, the National Cancer Institute is actively investing in building the infrastructure for CER in GM developing a roadmap for research and knowledge synthesis and evaluation.

      National Cancer Institute. Request for Applications in comparative effectiveness research in genomic and personalized medicine. Available at: Accessed May 22 2009.

      The emerging clinical and population health research infrastructure and new methods of knowledge generation and synthesis will, in turn, benefit and support the development of evidence from CER to aid decision making in 21st century medicine and public health.

      Additional information

      The views expressed in this paper do not necessarily reflect the views of the Department of Health and Human Services in the United States.
      Disclosure: The authors declare no conflict of interest.