Improved provider preparedness through an 8-part genetics and genomic education program

Published:November 30, 2021DOI:https://doi.org/10.1016/j.gim.2021.08.008

      Abstract

      Purpose

      Large-scale genetics education appropriate for general practice providers is a growing priority. We describe the content and impact of a mandatory system-wide program implemented at Sanford Health.

      Methods

      The Imagenetics Initiative at Sanford Health developed a 2-year genetics education program with quarterly web-based modules that were mandatory for all physicians and advanced practice providers. Scores of 0 to 5 were calculated for each module on the basis of the number of objectives that the participants reported as fulfilled. In addition, the participants completed surveys before starting and after finishing the education program, which included a 7-item measure scored 7 to 28 on the perceived preparedness to practice genetics.

      Results

      Between 2252 and 2822 Sanford Health employees completed each of the 8 quarterly education modules. The ratings were highest for the module about using genomics to improve patient management (mean score = 4.3) and lowest for the module about different types of genetic tests and specialists. The mean perceived preparedness scores increased from 15.7 at pre-education to 19.1 at post-education (P < .001).

      Conclusion

      Web-based genetics education was highly effective in increasing health care providers’ confidence about using genetics. Both comfort with personal knowledge and confidence regarding access to the system’s genomic medicine experts increased significantly. The results demonstrate how scalable approaches can improve provider preparedness.

      Graphical abstract

      Keywords

      Introduction

      The role of genetic testing in all aspects of medicine continues to increase rapidly,
      • Phillips K.A.
      • Deverka P.A.
      • Hooker G.W.
      • Douglas M.P.
      Genetic test availability and spending: where are we now? Where are we going?.
      but the number of genetic specialists is inadequate to meet current demands.
      • Kurnat-Thoma E.
      Educational and ethical considerations for genetic test implementation within health care systems.
      To realize the potential of genomic medicine, health care providers (HCPs) of all specialties, including those not trained in genetics, must be prepared to receive and act on genetic information.
      • Dragojlovic N.
      • Borle K.
      • Kopac N.
      • et al.
      The composition and capacity of the clinical genetics workforce in high-income countries: a scoping review.
      Nongenetic HCPs, however, consistently report poor knowledge and low confidence about using genetic test results in the care of their patients, particularly for those in primary care.
      • Nippert I.
      • Harris H.J.
      • Julian-Reynier C.
      • et al.
      Confidence of primary care physicians in their ability to carry out basic medical genetic tasks-a European survey in five countries-Part 1.
      • Mainous 3rd, A.G.
      • Johnson S.P.
      • Chirina S.
      • Baker R.
      Academic family physicians’ perception of genetic testing and integration into practice: a CERA study.
      • Carroll J.C.
      • Makuwaza T.
      • Manca D.P.
      • et al.
      Primary care providers’ experiences with and perceptions of personalized genomic medicine.
      • Harding B.
      • Webber C.
      • Ruhland L.
      • et al.
      Primary care providers’ lived experiences of genetics in practice.
      • Mikat-Stevens N.A.
      • Larson I.A.
      • Tarini B.A.
      Primary-care providers’ perceived barriers to integration of genetics services: a systematic review of the literature.
      • Scheuner M.T.
      • Sieverding P.
      • Shekelle P.G.
      Delivery of genomic medicine for common chronic adult diseases: a systematic review.
      Moreover, educational efforts often neglect the needs of HCPs such as nurses and advanced practice providers (APPs).
      • Campion M.
      • Goldgar C.
      • Hopkin R.J.
      • Prows C.A.
      • Dasgupta S.
      Genomic education for the next generation of health-care providers.
      ,
      • Crellin E.
      • McClaren B.
      • Nisselle A.
      • Best S.
      • Gaff C.
      • Metcalfe S.
      Preparing medical specialists to practice genomic medicine: education an essential part of a broader strategy.
      It is imperative for health systems to develop scalable strategies to engage, educate, and empower nongenetics HCPs in large numbers to use genetic information.
      • Campion M.
      • Goldgar C.
      • Hopkin R.J.
      • Prows C.A.
      • Dasgupta S.
      Genomic education for the next generation of health-care providers.
      ,
      Secretary’s Advisory Committee on Genetics, Health, and Society
      Genetics Education and Training. Department of Health & Human Services.
      ,
      • Rubanovich C.K.
      • Cheung C.
      • Mandel J.
      • Bloss C.S.
      Physician preparedness for big genomic data: a review of genomic medicine education initiatives in the United States.
      Most studies about genomics education for nongenetic HCPs are encouraging, which show improvements in knowledge, self-efficacy, and attitudes of the HCPs.
      • Talwar D.
      • Tseng T.S.
      • Foster M.
      • Xu L.
      • Chen L.S.
      Genetics/genomics education for nongenetic health professionals: a systematic literature review.
      Only a few of these data were derived from larger studies and over longer periods. Data from 143 general practitioners who completed a 2-year program showed substantial improvement in knowledge.
      • Clyman J.C.
      • Nazir F.
      • Tarolli S.
      • Black E.
      • Lombardi R.Q.
      • Higgins J.J.
      The impact of a genetics education program on physicians’ knowledge and genetic counseling referral patterns.
      Similarly, a 1-year program that included 8 or more hours of teaching, supplemented by written materials, showed improved knowledge and attitudes toward genetic services in 121 primary care providers.
      • Kolb S.E.
      • Aguilar M.C.
      • Dinenberg M.
      • Kaye C.I.
      Genetics education for primary care providers in community health settings.
      Pilot test data from a hybrid program of web-based modules and face-to-face lectures also showed improvements in knowledge.
      • Wallen G.R.
      • Cusack G.
      • Parada S.
      • Miller-Davis C.
      • Cartledge T.
      • Yates J.
      Evaluating a hybrid web-based basic genetics course for health professionals.
      Whether such improvements would be observed in larger, more sustained efforts that include nonvolunteers is unclear.
      Here, we summarize a mandatory 2-year provider education program implemented at Sanford Health. We summarize feedback regarding how well the modules increased participants’ self-reported knowledge, competence, and performance. We describe how effectively the modules met their stated objectives, and we identify factors that influenced ratings. The goal of this report is to offer guidance to health systems developing genetic education programs that are appropriate to the needs of providers who are not genetic specialists.

      Materials and Methods

       Overview

      Sanford Health serves more than 2 million patients through more than 2500 HCPs. In 2014, Sanford Health launched the Imagenetics (internal medicine + genetics) Initiative to accelerate the integration of genetics into patient care system-wide.
      • Christensen K.D.
      • Bell M.
      • Zawatsky C.L.B.
      • et al.
      Precision population medicine in primary care: the Sanford Chip experience.
      A goal of the Imagenetics Initiative is to increase HCPs’ preparedness to manage genetic findings, with an emphasis on the needs of general practice providers.
      A multimodal education plan was launched in 2017. Between 2017 and 2019, the quarterly educational program that all physicians and APPs are required to complete included 8 novel computer-based training modules. The modules included a combination of text, case vignettes, embedded videos, infographics, and recorded lectures released quarterly. Medical residents and fellows were not required to complete the modules. The modules were also available to other Sanford Health HCPs and administrators. The overarching goals of the modules were to (1) increase awareness and baseline knowledge of genomic medicine, (2) increase comfort using genomic medicine in routine clinical care, and (3) increase the understanding of when and how to access genetics specialists within the Sanford Health System.

       Development and administration of educational modules

      Existing provider education programs were considered, but they could not be obtained and adapted to the needs of Sanford Health because of licensing restrictions. Thus, the module content was internally developed. The scope and sequence of learning objectives were established by an expert leadership panel of subject-matter experts, including medical geneticists, clinical pharmacists with training in pharmacogenomics (PGx), laboratory directors, and genetic counselors. The learning objectives were refined by smaller groups. The format of each web-based presentation was determined on the basis of the content and varied from voiceover lectures to interactive modules that provided access to external resources. The committees drafted content for approval from Imagenetics clinical leadership. Approved content was then sent to an internal learning, education, and development team for implementation. Content, including module-specific objectives, was also sent to the continuing medical education (CME) office for review, and providers could earn CME credits for completing each of 6 of 8 modules. Two modules addressing the genetics of drug response and genetic screening using the Sanford Chip were not granted CME credit. The CME-granting committee felt that because these modules focused on specific tests offered from the Sanford Medical Genetics Laboratory, they were not free of commercial bias and therefore did not qualify for CME credit.
      The modules were distributed through an internal education portal that was used regularly for all mandatory provider educational programs and tracked the amount of time individuals spent on each module. Genetics education targeted to internists existed in the educational portal during the period of interest, although it was not promoted during the period of interest. The modules are available in the educational portal for providers to review as often as desired. Participation in each module was tied to compensation, with the providers being given 3 months to complete each successive module.

       Content of modules

      The content and objectives of each module are summarized in Table 1. The first series of 4 modules provided an overview of genomic medicine. The first module served as a foundation to help individuals better understand genomic medicine and how it can impact clinical practice. The content included how to recognize red flags for genetic disorders and an overview of how genes may affect responses to medications. The second module aimed to help individuals recognize genomic applications of precision medicine and the potential transformative effect on patient care. The components involved traditional applications, including analyses of the family histories of diseases. The module also addressed emerging applications such as genomic risk assessments for common diseases and provided an overview of the Sanford Chip Program. The third module focused on PGx and drug metabolism in the context of prodrug vs active drug. The content included where to find the guidelines for PGx testing and the use of clinical decision support tools. In the final module of the first year, the participants learned the difference between somatic and germline genetic testing. Various types of genetic testing options and their clinical applications were addressed, as were the roles of genetics professionals such as genetic counselors.
      Table 1Content and objectives of individual modules
      Module and DatesMinutes to Complete, Median (IQR)CME CreditOverviewLearning Objectives
      Series 1 (2017-2018): Genomic medicine in the clinical setting
       Module 1
      Combination of recorded lecture and interactive format.
      ,
      Genetic principles content.
      : What is genomic medicine?Jul-Sept 2017
      26 (38)YesFoundational material to better understand genomic medicine and how it can impact clinical practice
      • Describe genomic medicine
      • Interpret a pedigree
      • Differentiate between genotype and phenotype
      • Determine when to refer a patient for a genetic medicine consult
      • Describe PGx and its benefits
       Module 2
      Genetic principles content.
      ,
      Recorded lecture format.
      : Current applications of genomic medicineOct-Dec 2017
      21 (29)YesDescriptions of genomic applications of precision medicine, including preemptive genomic screening
      • Recognize genomic applications of precision medicine
      • Classify the components of genomic medicine
      • Describe the Sanford Chip and its clinical utility
      • Recognize the strengths and limitations of preemptive precision medicine
       Module 3
      Recorded lecture format.
      ,
      Combination of general principles and “how-to” content.
      : The genetics of drug response Jan-Mar 2018
      14 (20)NoClinical utility and application of PGx testing and the basics of drug metabolism
      • Define PGx metabolizer types in the context of prodrug versus active drugs
      • Identify the scientific organizations that create the guidelines for clinical application of PGx
      • Recognize the components of the PGx test and utilize decision support tools
      • Order PGx testing and apply results
       Module 4
      Recorded lecture format.
      ,
      Combination of general principles and “how-to” content.
      : Different types of genetic tests and specialistsApr-Jun 2018
      22 (33)YesExamples of genetic variation and types of tests used to identify each, along with types of genetic specialists available to offer support
      • Differentiate between somatic and germline variation
      • Summarize the different types of genetic testing
      • Recognize the clinical application for each type of genetic testing
      • Examine the clinical relevance of the genetic counseling process
      • Distinguish the difference between genetic professionals
      Series 2 (2018-2019): Clinical applications of genomic medicine
       Module 5
      Interactive format.
      ,
      “How-to” content.
      : PGx in patient careJul-Sept 2018
      25 (30)YesPGx principles review and common case examples showcasing available clinical decision support tools (recorded video lecture)
      • Apply the principles of PGx to patient care
      • Recognize cases in which PGx testing is appropriate
      • Discuss the advantages and disadvantages of current approaches to PGx testing
      • Recognize the components of PGx reports and utilize decision support tools
      • Identify clinical resources related to PGx testing
       Module 6
      Genetic principles content.
      ,
      Interactive format.
      : The spectrum of genetic variantsOct-Dec 2018
      16 (16)YesComparison of the genetics of mendelian and common diseases with an introduction to the identification and analysis of single-nucleotide variants (SNVs) (recorded video lecture)
      • Summarize past efforts and current opportunities related to precision medicine
      • Outline the spectrum of genetic changes, or variants, between mendelian inheritance and common disease
      • Characterize SNVs
      • Appreciate the design and clinical utility of genome-wide association studies (GWAS)
      • Assess the clinical application of polygenic risk scores (PRS) to modify patients’ clinical risk categories for more precise screening and treatment
       Module 7
      Recorded lecture format.
      ,
      “How-to” content.
      : Genetic screening and the Sanford ChipJan-Mar 2019
      8 (12)NoHigh-level overview of Imagenetics with a focus on the return of results workflow for Sanford’s precision prevention tool, the Sanford Chip
      • Describe the three main initiatives of Sanford Imagenetics with an emphasis on the Sanford Chip
      • Delineate the Sanford Chip workflow for return of results
      • Apply Sanford Chip results to clinical practice through case examples
       Module 8
      Recorded lecture format.
      ,
      “How-to” content.
      : Using genomics to improve managementApr-Jun 2019
      13 (15)YesCase examples outlining how a genetic diagnosis improves patient outcomes and a brief description of Sanford’s rare disease registry
      • Describe how a genetic diagnosis can aid patient care
      • Evaluate cases in which referring a patient for a genetic medicine consult may be valuable
      • Apply PGx testing results to medical management
      • Discuss the value that the Coordination of Rare Diseases at Sanford (CoRDS) provides to patients, families, and researchers
      Apr, April; CME, continuing medical education; IQR, interquartile range; Jan, January; Jul, July; Jun, June; Mar, March; Oct, October; PGx, pharmacogenomics; Sept, September.
      a Combination of recorded lecture and interactive format.
      b Genetic principles content.
      c Recorded lecture format.
      d Combination of general principles and “how-to” content.
      e Interactive format.
      f “How-to” content.
      The second series of 4 educational modules addressed more specific clinical applications. The first module of the second year focused primarily on the clinical application of PGx testing. Through a recorded lecture format, the participants learned when to order PGx testing and the pros and cons of current approaches. The module also reviewed components of a PGx report and expanded on how to utilize available decision support resources. The second module was also a recorded lecture and addressed the association between different types of genetic variants, mendelian inheritance, and common disease. The module also provided details about single-nucleotide variants, how they are identified, and how they may be used to estimate the polygenic risk for common diseases. The third module addressed the Sanford Chip Program, an elective genetic test that provides preemptive PGx testing and screens individuals for medically actionable genetic predispositions. Using case studies, the module provided examples of the application of the Sanford Chip to patient care as well as workflow for the return of results to providers and patients. The final module explained how a genetic diagnosis could improve patient care in a series of case studies. The content reviewed how to identify situations where a consult with a genetic specialist is warranted and how PGx findings could influence medical management. The module concluded with a discussion of the rare disease registry at Sanford Health.

       Measures of effectiveness

      Outcome data were analyzed from all available anonymous surveys administered to educational program participants via SurveyMonkey (Momentive, Inc.) for quality improvement purposes. Perceptions about preparedness, access to genetic specialists, and utility were queried rather than more objective measures, such as genetic knowledge scales, to minimize participant burden and because the content of year 2 modules had not been finalized at the time the program was launched.

       Assessments of individual modules

      After the completion of each of the 8 educational modules, the participants provided yes or no responses to the following statements: (1) “The content of this activity matched my current (or potential) scope of practice;” (2) “This activity increased my knowledge (knowing what to do);” (3) “This activity increased my competence (knowing how to do something);” and (4) “This activity improved my performance (ones actual behavior in practice).” All modules except the last also include a fifth statement, “Were your personal objectives successfully achieved?”

       Pre- and post-education assessment

      Before starting the first educational module and after completing the final one, surveys included the measures mentioned in the later sections.

       Perceived preparedness

      The individuals were asked to rate how prepared they felt about genetics in medicine in a set of 7 items: (1) I feel well-informed about knowledge of genetics; (2) I feel well-informed about genetic testing in general; (3) I feel comfortable ordering a genetic test to genetic conditions in my specialty; (4) I feel comfortable ordering a genetic test for disease susceptibility (eg, BRCA1/BRCA2 testing for the risk of breast and ovarian cancer); (5) I feel comfortable ordering a pharmacogenetic test to predict risk of adverse events or likelihood of response (eg, CYP2C9/VKORC1 and warfarin therapy); (6) I have access to genetics expertise when I have a question related to a patient; and (7) I feel that my medical training adequately prepared me to appropriately order and use genetics tests. The response options of “strongly disagree,” “disagree,” “agree,” and “strongly agree” were scored 1 to 4, respectively, and summed to create a summary score of 7 to 28, with higher scores indicating stronger feelings of perceived preparedness. The scale items were novel but demonstrated strong internal consistency (Cronbach alpha of 0.91 at pre-education and 0.93 at post-education).

       Perceived access to genetic specialists

      Individuals were asked separate yes or no questions about whether they had a geneticist or genetic counselor to whom they could refer patients.

       Perceived utility

      Individuals were also asked how useful they thought pharmacogenetic results would be for managing their patients’ health, with response options of “very useful,” “somewhat useful,” “not very useful,” and “not at all useful.”

       Open-ended items

      The respondents provided open-ended feedback to the following 2 questions on both pre- and post-education questionnaires: “How do you feel about genetic testing becoming part of routine clinical care?” and “What additional information would be helpful to increase your comfort with using genetics in clinical care?”

       Respondent characteristics

      Respondents self-reported their gender; their age in 10-year increments; role (Physician, APP, Nursing, Pharmacy, other); specialty; and, if applicable, years out of residency, training (US or non-US medical school), and residency training setting (university-based, hospital-based, other).

       Data analysis

      The analyses of pre- and post-education surveys were limited to data from physicians and APPs because these providers were the target audience and were required to complete the modules. Module-specific ratings included all survey completers because physicians and APPs could not be distinguished from individuals with other roles. To avoid instances where outcomes were provided long after the completion of education modules, data were analyzed from surveys we could match to records confirming module completion within 14 days. We also omitted 462 respondents from analyses for pre-education surveys who reported that they had already completed the required Sanford training modules.
      Chi-square and Wilcoxon rank sum tests were used to compare the characteristics of the respondents of the pre- and post-education assessments. Linear and logistic regression models were used to compare pre- and post-education survey data as appropriate to the distributions of responses. Covariates included gender, age, and role. Generalized linear models with logit links and binomial distributions were also used to compare module-specific evaluation data by topic, although no respondent characteristics were included in these statistical models because the surveys were anonymous. We also analyzed module-specific education descriptively by content and format. The content of individual modules was classified using the frameworks proposed for genetic literacy
      • Smerecnik C.M.R.
      • Mesters I.
      • de Vries N.K.
      • de Vries H.
      Educating the general public about multifactorial genetic disease: applying a theory-based framework to understand current public knowledge.
      ,
      • Smerecnik C.M.
      • Mesters I.
      • de Vries N.K.
      • de Vries H.
      Applying a theory-based framework to understand public knowledge of genetic risk factors: a case for the distinction between how-to knowledge and principles knowledge.
      as primarily principles knowledge (underlying theoretical principles of genetics and medical genetics), “how-to” knowledge (practical knowledge concerning the proper use of genetic testing), or both. The format of the individual modules was classified as interactive, recorded presentation, or both.
      Open-ended data were classified using approaches developed for coding qualitative data.
      • Strauss A.
      • Corbin J.
      Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory.
      First, 1 study team member (K.D.C.) proposed an initial codebook based on the review of responses. Two study team members (L.N.G. and C.L.P.) then coded each response set independently. In instances where interrater reliability metrics were suboptimal, the codebooks were revised, and data were recoded until agreement was strong (Cohen κ > 0.8). The final differences in coding were reconciled by a single study team member (L.N.G.).
      Available-case analyses were conducted using R version 4.0.3 (R Foundation for Statistical Computing). The study was deemed exempt from human subjects research by the Sanford Research Institutional Review Board.

      Results

       Response rates and provider characteristics

      Between 2252 and 2822 individuals completed each module, and between 1263 and 2377 individuals completed each postmodule assessment (Supplemental Table 1 and Supplemental Table 2). The average response rates to module-specific surveys decreased from 95.3% after completion of the first module to 50.9% after completion of the final module. The median time required to complete each module ranged from 8 to 26 minutes (Table 1), with participants spending an average of 221 minutes (SD = 151 minutes) overall.
      Pre- and post-education assessments had comparable completion percentages from physicians and APPs. At pre-education, 1719 of 2102 physicians and APPs (81.8%) completed the pre-education survey, and 1263 of 2482 physicians and APPs (50.9%) completed the post-education survey. The characteristics of physicians and APPs who completed the pre-education and post-education surveys are summarized in Table 2. The respondents analyzed from the post-education survey were more likely to be APPs (38.8% vs 43.1%, respectively, P = .019), were older (58.2% over the age of 40 vs 65.1%, respectively, P < .001), and were more likely to report receiving genetics education in medical school or during residency (52.4% vs 64.1%, respectively, P < .001) than respondents of the pre-education survey. The physicians who completed the post-education survey tended to be more experienced (51.7% at least 10 years out of residency vs 64.9%, respectively, P < .001) than the physicians who completed the pre-education survey.
      Table 2Characteristics of pre-education and post-education survey respondents
      CharacteristicPre-Education Survey, n (%)Post-education Survey, n (%)P
      Role.019
       Physician1052 (61.2)719 (56.9)
       APP667 (38.8)544 (43.1)
      Age, y<.001
       <3097 (5.6)54 (4.3)
       30-39618 (36.0)385 (30.5)
       40-49428 (24.9)343 (27.2)
       50-59304 (17.7)254 (20.1)
       60-69225 (13.1)197 (15.6)
       70+38 (2.2)24 (1.9)
       Missing age9 (0.5)6 (0.5)
      Primary specialty
      At pre-education, respondents could endorse multiple primary specialties and write in others. At post-education, respondents could choose a single response option and write in others. Additional specialties reported are summarized in Supplemental Table 2.
       Family medicine383 (22.3)274 (21.7)
       Internal medicine137 (8.0)90 (7.1)
       OB/Gyn76 (4.4)43 (3.4)
       Pediatrics/Pediatric subspecialties156 (9.1)99 (7.8)
      Years out of residency
      Data for these items were collected only from physicians.
      <.001
       <5289 (27.5)113 (15.7)
       5-9208 (19.8)130 (18.1)
       10-14136 (12.9)99 (13.8)
       15-19131 (12.5)87 (12.1)
       20+266 (25.3)263 (36.6)
      Out of residency missing22 (2.1)27 (3.8)
      Genetics education in medical school or residency900 (52.4)810 (64.1)<.001
      Evaluations of specific modules did not collect information about respondent characteristics.
      APP, advanced practice provider; OB/Gyn, obstetrics/gynecology.
      a At pre-education, respondents could endorse multiple primary specialties and write in others. At post-education, respondents could choose a single response option and write in others. Additional specialties reported are summarized in Supplemental Table 2.
      b Data for these items were collected only from physicians.

       Postmodule assessments

      Most individuals who completed assessments judged the modules’ formats to be satisfactory. Between 77.1% and 92.6% of respondents reported that the format of each module was appropriate, and when queried in year 2, over 96% of participants said that the length of each module was appropriate (Supplemental Table 3). When improvements to the format of modules were suggested, the respondents asked for case-based presentations more often than other changes.
      Assessments of whether individual modules met goals are summarized in Table 3. The percentages of respondents who reported that the modules increased their knowledge increased their competence, and improved their performance were lowest for the modules that addressed different types of genetic tests and specialists and current applications of genomic medicine. In contrast, the percentages of respondents who reported that the same module goals (knowledge, competence, performance) were achieved were highest for the modules about using genomics to improve management, the module about genetic screening and the Sanford Chip, and the module about PGx in patient care.
      Table 3Percentage of respondents who reported that educational modules achieved goals
      ModuleIncreased Knowledge, %Increased Competence, %Improved Performance, %Matched Scope of Practice, %Personal Objectives Met, %
      1. What is genomic medicine? (n = 1914-2043)86.576.163.688.191.8
      2. Current applications of genomic medicine (n = 1687-1960)87.376.460.983.186.1
      3. The genetics of drug response (n = 1635-2049)90.382.165.785.089.6
      4. Different types of genetic tests and specialists (n = 1224-1292)84.171.554.774.779.9
      5. PGx in patient care (n = 1471-1699)93.087.973.479.787.0
      6. The spectrum of genetic variants (n = 1445-1643)89.176.465.574.188.1
      7. Genetic screening and the Sanford Chip (n = 2063-2370)92.587.573.279.490.5
      8. Using genomics to improve management (n = 1211-1233)93.386.974.380.8
      Item was not included in the module-specific survey.
      PGx, pharmacogenomics.
      a Item was not included in the module-specific survey.
      Analyses suggested that the respondents preferred modules that focused on practical, “how-to” knowledge. For the 3 modules where the content was primarily “how-to” knowledge, 92.9% of respondents reported increased knowledge, 87.4% reported increased competence, and 73.6% reported improved performance (Supplemental Table 4). In contrast, across the other 5 modules, 87.5% reported increased knowledge, 76.7% reported increased competence, and 62.3 reported improved performance (all P < .001). Interestingly, analyses showed little difference in the respondents’ ratings of modules that primarily used either an interactive or a recorded format (Supplemental Table 5).

       Pre- and post-education assessments

      The mean scores on our 7 to 28 scale of perceived preparedness increased from 15.7 at pre-education to 19.2 at post-education (difference (diff) = 3.5, 95% confidence interval [CI] = 3.2-3.8, P < .001). Regression analyses are summarized in Table 4. Changes were particularly large among APPs, where the mean scores increased from 14.2 to 18.8 (diff = 4.6, 95% CI = 4.1-5.1, P < .001). In contrast, the mean scores among physicians increased from 16.7 to 19.4 (diff = 2.7, 95% CI = 2.3-3.1, P < .001). Perceived preparedness scores varied by age across time points, with an average decrease of 0.3 points on the scale per 10-year increase in age (95% CI = –0.4 to –0.2, P < .001). We also observed lower scores at both time points among older respondents.
      Table 4Summary of linear regression analyses of perceived preparedness scores (7-28 scale)
      VariableEstimateStandard ErrorP
      Intercept17.440.39<.001
      Male gender (ref: female)–0.260.18.156
      Age in y (ref: <30)<.001
       30-39–0.280.37
       40-49–0.640.38
       50-59–0.980.39
       60-69–1.150.41
       70+–0.520.64
      APP role (ref: physician)–2.500.23<.001
      Change among physicians, post-education minus pre-education2.670.20<.001
      Change among APPs, post-education minus pre-education1.960.32<.001
      Model estimates represent the difference in scale scores compared with the reference group.
      APP, advanced practice provider; ref, reference.
      Increases on all items of the perceived preparedness scale were observed from pre-education to post-education (Figure 1). The largest increases were observed on the percentage of participants who reported feeling well-informed about their knowledge of genetics (36% agreed at pre-education vs 81% agreed at post-education) and the percentage of participants who reported feeling comfortable ordering PGx testing (19% agreed at pre-education vs 60% agreed at post-education). At pre-education, the respondents were least likely to report feeling comfortable ordering PGx testing (19% agreed), whereas at post-education, the respondents were least likely to report that their medical training adequately prepared them to order and use genetic tests (55% agreed). At both time points, the respondents were most likely to report that they had access to genetics expertise when they had questions about a patient (59% agreed at pre-education, 86% agreed at post-education).
      Figure thumbnail gr1
      Figure 1Percentage of physicians and advanced practice providers who agreed or strongly agreed with each item of the perceived preparedness scale. Percentages were estimated using logistic regression equations, with adjustment for role, age, and gender.
      We also observed an increase in the proportion of physicians and APPs who reported having access to a geneticist or genetic counselor to whom they could refer patients when asked as a yes or no question. In total, 64.9% of physicians and APPs reported access to one or both of these specialists at pre-education compared with 82.7% at post-education (P <.001). Increases were also observed in the perceived utility of PGx testing. At pre-education, 22.2% of respondents reported that PGx results would be “very useful” for managing their patients’ health, compared with 37.8% of respondents at post-education (P < .001).
      Analyses of open-ended items also showed a lower likelihood of addressing further education as helpful in the post-education survey than in the pre-education survey. The odds that the respondents’ written responses about what would increase their comfort with genetics-addressed education decreased by 45.6% after education than before education (P = .003). Interestingly, the odds that the respondents’ written responses were about the need for more experience or practice were 4.95 times higher at post-education than at pre-education (P < .001).

      Discussion

      The Sanford Health experience is one of the first examples of a sustained, mandatory genetics education program at a major health system. Over 2000 providers completed the program over a 2-year period, and the program yielded significant improvement in provider preparedness, including large increases in provider confidence, awareness of help, and perceived utility of genetic testing. A large majority of individuals reported that the modules increased their knowledge and competence and that their personal objectives were met. The findings from this program demonstrate how committed health systems can effectively provide genetics education to their HCPs as part of a comprehensive plan to implement genomic medicine.
      Importantly, whereas substantial investment was necessary to create the educational modules, program completion did not appear to be a burden. Well over 90% of individuals reported that the length of each of the modules was appropriate when that question was included in module assessments. Efforts to educate providers about genetics have varied greatly in their time demands, ranging from half-day courses to multiday seminars to monthly meetings over a year.
      • Manolio T.A.
      • Chisholm R.L.
      • Ozenberger B.
      • et al.
      Implementing genomic medicine in the clinic: the future is here.
      • Demmer L.A.
      • Waggoner D.J.
      Professional medical education and genomics.
      • Carroll J.C.
      • Wilson B.J.
      • Allanson J.
      • et al.
      GenetiKit: a randomized controlled trial to enhance delivery of genetics services by family physicians.
      • Carroll J.C.
      • Rideout A.L.
      • Wilson B.J.
      • et al.
      Genetic education for primary care providers: improving attitudes, knowledge, and confidence.
      • Blazer K.R.
      • Christie C.
      • Uman G.
      • Weitzel J.N.
      Impact of web-based case conferencing on cancer genetics training outcomes for community-based clinicians.
      Such efforts are often difficult for providers to accommodate in their schedules. Organizations such as the Jackson Laboratory, the Centers for Disease Control and Prevention, and the International Society of Nurses in Genetics have developed and curated web-based provider education programs that provide greater scheduling flexibility.
      Clinical education at Jax. The Jackson Laboratory.
      Training programs and courses. Centers for Disease Control and Prevention.
      Online resources for health professionals. International Society of Nurses in Genetics.
      The Sanford Health program has taken these efforts a step further by providing web-based programs that individuals can complete at their own pace, but have been tailored to the services and health care provider support infrastructure developed by Sanford’s Imagenetics Initiative.
      One of the key goals of Sanford’s genomics educational program was to make providers who may have little experience with genetics more comfortable with population preemptive genetic screening. The modules focused on the use of genetic information in the care of both healthy and sick patients as well as the benefits and limitations of genetic testing. The percentage of individuals who reported that the modules increased their competence and performance was more than 10% higher when the modules emphasized “how-to” knowledge or case examples rather than genetic principles. In particular, the module about genetic screening and the Sanford Chip was among the highest-rated even though it did not qualify for CME credit. The findings from our work add weight to calls for a greater use of theoretical frameworks and educational theory to inform program development by demonstrating how content may affect responses to genetic education programs.
      • Talwar D.
      • Tseng T.S.
      • Foster M.
      • Xu L.
      • Chen L.S.
      Genetics/genomics education for nongenetic health professionals: a systematic literature review.
      ,
      • Guttmacher A.E.
      • Porteous M.E.
      • McInerney J.D.
      Educating health-care professionals about genetics and genomics.
      • Burke W.
      • Emery J.
      Genetics education for primary-care providers.
      • Reed E.K.
      • Johansen Taber K.A.
      • Ingram Nissen T.
      • et al.
      What works in genomics education: outcomes of an evidenced-based instructional model for community-based physicians.
      Although it is important to include principles knowledge, doing so in a manner that also includes “how-to” content may yield the best results, especially if it is linked to a high-profile program that could affect the practice of most participants.
      • Smerecnik C.M.R.
      • Mesters I.
      • de Vries N.K.
      • de Vries H.
      Educating the general public about multifactorial genetic disease: applying a theory-based framework to understand current public knowledge.
      ,
      • Smerecnik C.M.
      • Mesters I.
      • de Vries N.K.
      • de Vries H.
      Applying a theory-based framework to understand public knowledge of genetic risk factors: a case for the distinction between how-to knowledge and principles knowledge.
      One factor that may have increased the effectiveness of the genetics education program was a significant effort to increase provider support for and awareness of the role for genetics in medicine before the launch of the formal educational program. Experts generally agree that education alone may be insufficient to ensure the appropriate use of genetic testing.
      • Gaff C.L.
      • Winship I.M.
      • Forrest S.M.
      • et al.
      Preparing for genomic medicine: a real world demonstration of health system change.
      ,
      • Bennett C.L.
      • Burke S.E.
      • Burton H.
      • Farndon P.A.
      A toolkit for incorporating genetics into mainstream medical services: learning from service development pilots in England.
      The efforts at Sanford Health included infrastructure development for the integration of genetic information into the electronic medical record (EMR) and development of automated clinical decision support that would provide point-of-care guidance to providers. The educational modules that addressed “how-to” content leveraged the content of the existing infrastructure. In a parallel effort, Sanford Health increased the number of genetic counselors in its system and embedded them in all internal medicine clinics. The combination of the “human” resource with support in the EMR may have enhanced educational efforts by making providers more aware that help for responding to genetic information was readily available.
      • Sasaki N.
      • Yamaguchi N.
      • Okumura A.
      • et al.
      Factors affecting the use of clinical practice guidelines by hospital physicians: the interplay of IT infrastructure and physician attitudes.
      ,
      • Trinidad S.B.
      • Fryer-Edwards K.
      • Crest A.
      • Kyler P.
      • Lloyd-Puryear M.A.
      • Burke W.
      Educational needs in genetic medicine: primary care perspectives.
      Notably, the genetic education program overlapped with the launch of the Sanford Chip in 2018, which is a flagship genetic testing program in primary care settings that offers pharmacogenomic testing and optional genetic risk information. Efforts to make providers and patients aware of this new elective service likely had a significant impact on provider awareness for the role of genetics in medicine and increased the salience of the genetics education program and engagement with the content. Importantly, other studies suggest that the providers may be unwilling to engage with genetic information if they feel inadequately prepared or supported.
      • Pet D.B.
      • Holm I.A.
      • Williams J.L.
      • et al.
      Physicians’ perspectives on receiving unsolicited genomic results.
      • Christensen K.D.
      • Bernhardt B.A.
      • Jarvik G.P.
      • et al.
      Anticipated responses of early adopter genetic specialists and nongenetic specialists to unsolicited genomic secondary findings.
      • Peterson J.F.
      • Field J.R.
      • Shi Y.
      • et al.
      Attitudes of clinicians following large-scale pharmacogenomics implementation.
      The development of an environment to manage genetic information and the launch of provider education in genetics before offering the Sanford Chip may be a reason that over 11,000 patients have participated in the program to date.
      One limitation of our study was the use of self-assessments of genomic readiness. Preliminary analyses of data from 2018 showed that the providers altered medication choices or patient monitoring in 45% of encounters where potential drug-gene interactions were identified, including in 59% of encounters involving clopidogrel. These rates are higher than those observed in related clinical trials
      • Tuteja S.
      • Glick H.
      • Matthai W.
      • et al.
      Prospective CYP2C19 genotyping to guide antiplatelet therapy following percutaneous coronary intervention: a pragmatic randomized clinical trial.
      and much higher than a 10% concordance rate with clinical decision support recommendations observed at Sanford Health overall. Future work will refine these analyses as well as examine more objective measures of provider knowledge and behavior. Limitations also include the analysis of anonymous data that did not allow comparisons of pre- and post-education responses for specific individuals. It is possible that individuals with more positive attitudes about genomics were more likely to complete the post-education survey. The program was implemented in a single health care system, and the results may not generalize well to others, particularly systems lacking the bioinformatics infrastructure and clinical decision support to complement the education.
      Despite the significant efforts described here, our analyses still suggest that the providers felt additional education would be helpful. This, along with the perception that genetics has promise for the future, demonstrates the need to supplement system-wide educational efforts. Additional programs that Sanford Health has implemented include developing brief educational PowerPoint presentations that are available on demand. These presentations address topics such as the meaning of uninformative findings and how to use Genomic Indicators, which are tools in the Epic EMR system that document genomic findings as discrete fields that can trigger automated decision support.
      Nevertheless, our work demonstrates that health systems can effectively deliver provider-directed genetic education at scale. The modules summarized here have been combined into a single module, which is updated as needed to ensure that content is current with evolving best practice recommendations. All new physicians and APP hires complete this single module during orientation. We intend to provide ongoing education to build upon this existing foundation and respond to the rapid speed at which genetics is impacting medicine.

      Data Availability

      Data and code will be made available on request. Inquiries can be directed to the corresponding author.

      Acknowledgments

      The following people contributed to the development of the educational modules: Aissa Aifaoui, Jordan Baye, Megan Bell, Laura Davis-Keppen, Kristen DeBerg, Catherine Hajek, Allison Hutchinson, Patricia Crotwell Leiferman, Amanda Massmann, Lisa Mullineaux, Natasha Petry, Dylan Platt, April Schultz, and D. Isum Ward. This work was funded by the Sanford Health System. K.D.C. was supported by National Institutes of Health grants K01-HG009173 and R01-HD090019, and R.C.G. and C.L.B.Z. were also supported by National Institutes of Health grant R01-HL143295.

      Author Information

      Conceptualization: C.H., A.M.H., R.C.G., N.P., C.L.B.Z., K.D.C.; Data Curation: C.H., A.M.H., L.N.G., N.P., C.L.P., E.S.Z., K.D.C.; Formal Analysis: C.H., A.M.H., L.N.G., C.L.P., E.S.Z., K.D.C.; Funding Acquisition: C.H., R.C.G., K.D.C.; Investigation: C.H., A.M.H.; Methodology: C.H., A.M.H., L.N.G., N.P., C.L.P., C.L.B.Z., E.S.Z., K.D.C.; Project Administration: C.H., A.M.H., K.D.C.; Resources: R.G., K.D.C.; Supervision: C.H., A.M.H., L.N.G., K.D.C.; Validation: C.H., A.M.H., R.C.G., M.F.M., E.S.Z., K.D.C.; Visualization: L.N.G., K.D.C.; Writing-original draft: C.H., A.M.H., L.N.G., N.P., C.L.B.Z., K.D.C.; Writing-review and editing: C.H., A.M.H., L.N.G., R.C.G., M.F.M., N.P., C.L.P., C.L.B.Z., E.S.Z., K.D.C.

      Ethics Declaration

      The study was deemed exempt from human subjects research by the Sanford Research Institutional Review Board.

      Conflict of Interest

      Robert C. Green has received compensation for advising the following companies: AIA, Grail, Humanity, Kneed Media, Plumcare, UnitedHealth, Verily, Vibrent Health, and Wamberg and is the cofounder of Genome Medical, Inc, a technology and services company providing genetics expertise to patients, providers, employers, and care systems.

      Members of the Imagenetics METRICS Team

      Jordan Baye, Megan Bell, Kristen Deberg, Benjamin Forred, Colette Free, Catherine Hajek, Joel Van Heukelom, Ashley Hopp, Allison Hutchinson, Ryne Lees, Jennifer Leonhard, Amanda Massmann, Michelle Moore, Amelia Mroch, Natasha Petry, Dylan Platt, Erin Royer, April Schultz, Murat Sincan, Bethany Tucker, Elizabeth Wheeler; Pilgrim Health Care Institute: Kurt Christensen, Lauren Galbraith, Jessica LeBlanc, Ryan Walsh, and Emilie Zoltick; Robert Green, Charlene Preys, and Carrie Zawatsky; Lisa Mullineaux; Leila Jamal.
      Members of Imagenetics METRICS are included in the Supplementary Information.

      Additional Information

      References

        • Phillips K.A.
        • Deverka P.A.
        • Hooker G.W.
        • Douglas M.P.
        Genetic test availability and spending: where are we now? Where are we going?.
        Health Aff (Millwood). 2018; 37: 710-716https://doi.org/10.1377/hlthaff.2017.1427
        • Kurnat-Thoma E.
        Educational and ethical considerations for genetic test implementation within health care systems.
        Netw Syst Med. 2020; 3: 58-66https://doi.org/10.1089/nsm.2019.0010
        • Dragojlovic N.
        • Borle K.
        • Kopac N.
        • et al.
        The composition and capacity of the clinical genetics workforce in high-income countries: a scoping review.
        Genet Med. 2020; 22 (Published correction appears in Genet Med. 2020 published correction appears in Genet Med. 2020;22(9):1570): 1437-1449
        • Nippert I.
        • Harris H.J.
        • Julian-Reynier C.
        • et al.
        Confidence of primary care physicians in their ability to carry out basic medical genetic tasks-a European survey in five countries-Part 1.
        J Community Genet. 2011; 2: 1-11https://doi.org/10.1007/s12687-010-0030-0
        • Mainous 3rd, A.G.
        • Johnson S.P.
        • Chirina S.
        • Baker R.
        Academic family physicians’ perception of genetic testing and integration into practice: a CERA study.
        Fam Med. 2013; 45: 257-262
        • Carroll J.C.
        • Makuwaza T.
        • Manca D.P.
        • et al.
        Primary care providers’ experiences with and perceptions of personalized genomic medicine.
        Can Fam Physician. 2016; 62: e626-e635
        • Harding B.
        • Webber C.
        • Ruhland L.
        • et al.
        Primary care providers’ lived experiences of genetics in practice.
        J Community Genet. 2019; 10: 85-93https://doi.org/10.1007/s12687-018-0364-6
        • Mikat-Stevens N.A.
        • Larson I.A.
        • Tarini B.A.
        Primary-care providers’ perceived barriers to integration of genetics services: a systematic review of the literature.
        Genet Med. 2015; 17: 169-176https://doi.org/10.1038/gim.2014.101
        • Scheuner M.T.
        • Sieverding P.
        • Shekelle P.G.
        Delivery of genomic medicine for common chronic adult diseases: a systematic review.
        JAMA. 2008; 299: 1320-1334https://doi.org/10.1001/jama.299.11.1320
        • Campion M.
        • Goldgar C.
        • Hopkin R.J.
        • Prows C.A.
        • Dasgupta S.
        Genomic education for the next generation of health-care providers.
        Genet Med. 2019; 21: 2422-2430https://doi.org/10.1038/s41436-019-0548-4
        • Crellin E.
        • McClaren B.
        • Nisselle A.
        • Best S.
        • Gaff C.
        • Metcalfe S.
        Preparing medical specialists to practice genomic medicine: education an essential part of a broader strategy.
        Front Genet. 2019; 10: 789https://doi.org/10.3389/fgene.2019.00789
        • Secretary’s Advisory Committee on Genetics, Health, and Society
        Genetics Education and Training. Department of Health & Human Services.
        (2011.) (Accessed September 9, 2021.)
        • Rubanovich C.K.
        • Cheung C.
        • Mandel J.
        • Bloss C.S.
        Physician preparedness for big genomic data: a review of genomic medicine education initiatives in the United States.
        Hum Mol Genet. 2018; 27: R250-R258https://doi.org/10.1093/hmg/ddy170
        • Talwar D.
        • Tseng T.S.
        • Foster M.
        • Xu L.
        • Chen L.S.
        Genetics/genomics education for nongenetic health professionals: a systematic literature review.
        Genet Med. 2017; 19: 725-732https://doi.org/10.1038/gim.2016.156
        • Clyman J.C.
        • Nazir F.
        • Tarolli S.
        • Black E.
        • Lombardi R.Q.
        • Higgins J.J.
        The impact of a genetics education program on physicians’ knowledge and genetic counseling referral patterns.
        Med Teach. 2007; 29: e143-e150https://doi.org/10.1080/01421590701477373
        • Kolb S.E.
        • Aguilar M.C.
        • Dinenberg M.
        • Kaye C.I.
        Genetics education for primary care providers in community health settings.
        J Community Health. 1999; 24: 45-59https://doi.org/10.1023/a:1018765215470
        • Wallen G.R.
        • Cusack G.
        • Parada S.
        • Miller-Davis C.
        • Cartledge T.
        • Yates J.
        Evaluating a hybrid web-based basic genetics course for health professionals.
        Nurse Educ Today. 2011; 31: 638-642https://doi.org/10.1016/j.nedt.2010.11.001
        • Christensen K.D.
        • Bell M.
        • Zawatsky C.L.B.
        • et al.
        Precision population medicine in primary care: the Sanford Chip experience.
        Front Genet. 2021; 12: 626845https://doi.org/10.3389/fgene.2021.626845
        • Smerecnik C.M.R.
        • Mesters I.
        • de Vries N.K.
        • de Vries H.
        Educating the general public about multifactorial genetic disease: applying a theory-based framework to understand current public knowledge.
        Genet Med. 2008; 10: 251-258https://doi.org/10.1097/GIM.0b013e31816b4ffd
        • Smerecnik C.M.
        • Mesters I.
        • de Vries N.K.
        • de Vries H.
        Applying a theory-based framework to understand public knowledge of genetic risk factors: a case for the distinction between how-to knowledge and principles knowledge.
        Public Health Genomics. 2011; 14: 259-270https://doi.org/10.1159/000294149
        • Strauss A.
        • Corbin J.
        Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory.
        2nd ed. SAGE Publications, Thousand Oaks1998
        • Manolio T.A.
        • Chisholm R.L.
        • Ozenberger B.
        • et al.
        Implementing genomic medicine in the clinic: the future is here.
        Genet Med. 2013; 15: 258-267https://doi.org/10.1038/gim.2012.157
        • Demmer L.A.
        • Waggoner D.J.
        Professional medical education and genomics.
        Annu Rev Genomics Hum Genet. 2014; 15: 507-516https://doi.org/10.1146/annurev-genom-090413-025522
        • Carroll J.C.
        • Wilson B.J.
        • Allanson J.
        • et al.
        GenetiKit: a randomized controlled trial to enhance delivery of genetics services by family physicians.
        Fam Pract. 2011; 28: 615-623https://doi.org/10.1093/fampra/cmr040
        • Carroll J.C.
        • Rideout A.L.
        • Wilson B.J.
        • et al.
        Genetic education for primary care providers: improving attitudes, knowledge, and confidence.
        Can Fam Physician. 2009; 55: e92-e99
        • Blazer K.R.
        • Christie C.
        • Uman G.
        • Weitzel J.N.
        Impact of web-based case conferencing on cancer genetics training outcomes for community-based clinicians.
        J Cancer Educ. 2012; 27: 217-225https://doi.org/10.1007/s13187-012-0313-8
      1. Clinical education at Jax. The Jackson Laboratory.
      2. Training programs and courses. Centers for Disease Control and Prevention.
      3. Online resources for health professionals. International Society of Nurses in Genetics.
        https://www.isong.org/page-1325262
        Date accessed: January 10, 2021
        • Guttmacher A.E.
        • Porteous M.E.
        • McInerney J.D.
        Educating health-care professionals about genetics and genomics.
        Nat Rev Genet. 2007; 8: 151-157https://doi.org/10.1038/nrg2007
        • Burke W.
        • Emery J.
        Genetics education for primary-care providers.
        Nat Rev Genet. 2002; 3: 561-566https://doi.org/10.1038/nrg845
        • Reed E.K.
        • Johansen Taber K.A.
        • Ingram Nissen T.
        • et al.
        What works in genomics education: outcomes of an evidenced-based instructional model for community-based physicians.
        Genet Med. 2016; 18: 737-745https://doi.org/10.1038/gim.2015.144
        • Gaff C.L.
        • Winship I.M.
        • Forrest S.M.
        • et al.
        Preparing for genomic medicine: a real world demonstration of health system change.
        NPJ Genom Med. 2017; 2 (Published correction appears in NPJ Genom Med. 2017;2:31): 16
        • Bennett C.L.
        • Burke S.E.
        • Burton H.
        • Farndon P.A.
        A toolkit for incorporating genetics into mainstream medical services: learning from service development pilots in England.
        BMC Health Serv Res. 2010; 10: 125https://doi.org/10.1186/1472-6963-10-125
        • Sasaki N.
        • Yamaguchi N.
        • Okumura A.
        • et al.
        Factors affecting the use of clinical practice guidelines by hospital physicians: the interplay of IT infrastructure and physician attitudes.
        Implement Sci. 2020; 15: 101https://doi.org/10.1186/s13012-020-01056-1
        • Trinidad S.B.
        • Fryer-Edwards K.
        • Crest A.
        • Kyler P.
        • Lloyd-Puryear M.A.
        • Burke W.
        Educational needs in genetic medicine: primary care perspectives.
        Community Genet. 2008; 11: 160-165https://doi.org/10.1159/000113878
        • Pet D.B.
        • Holm I.A.
        • Williams J.L.
        • et al.
        Physicians’ perspectives on receiving unsolicited genomic results.
        Genet Med. 2019; 21: 311-318https://doi.org/10.1038/s41436-018-0047-z
        • Christensen K.D.
        • Bernhardt B.A.
        • Jarvik G.P.
        • et al.
        Anticipated responses of early adopter genetic specialists and nongenetic specialists to unsolicited genomic secondary findings.
        Genet Med. 2018; 20: 1186-1195https://doi.org/10.1038/gim.2017.243
        • Peterson J.F.
        • Field J.R.
        • Shi Y.
        • et al.
        Attitudes of clinicians following large-scale pharmacogenomics implementation.
        Pharmacogenomics J. 2016; 16: 393-398https://doi.org/10.1038/tpj.2015.57
        • Tuteja S.
        • Glick H.
        • Matthai W.
        • et al.
        Prospective CYP2C19 genotyping to guide antiplatelet therapy following percutaneous coronary intervention: a pragmatic randomized clinical trial.
        Circ Genom Precis Med. 2020; 13e002640https://doi.org/10.1161/CIRCGEN.119.002640