If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Preliminary validation of a consumer-oriented colorectal cancer risk assessment tool compatible with the US Surgeon General’s My Family Health Portrait
This study examines the analytic validity of a software tool designed to provide individuals with risk assessments for colorectal cancer based on personal health and family history information. The software is compatible with the US Surgeon General’s My Family Health Portrait (MFHP).
Methods
An algorithm for risk assessment was created using accepted colorectal risk assessment guidelines and programmed into a software tool (MFHP). Risk assessments derived from 150 pedigrees using the MFHP tool were compared with “gold standard” risk assessments developed by three expert cancer genetic counselors.
Results
Genetic counselor risk assessments showed substantial, but not perfect, agreement. MFHP risk assessments for colorectal cancer yielded a sensitivity for colorectal cancer risk of 81% (95% confidence interval: 54–96%) and specificity of 90% (95% confidence interval: 83–94%), as compared with genetic counselor pedigree review. The positive predictive value for risk for MFHP was 48% (95% confidence interval: 29–68%), whereas the negative predictive value was 98% (95% confidence interval: 93–99%). Agreement between MFHP and genetic counselor pedigree review was moderate (κ = 0.54).
Conclusion
The analytic validity of the MFHP colorectal cancer risk assessment software is similar to those of other types of screening tools used in primary care. Future investigations should explore the clinical validity and utility of the software in diverse population groups.
Patient-completed electronic family history (FH) collection tools have been proposed as a way to overcome several of the barriers to collection and use of FH information in health-care settings.
Several patient-completed tools designed to aid in the collection and interpretation of FH have been developed and validated, often in settings such as cancer and primary-care clinics.
A 2009 review of FH literature conducted for the US Agency for Health Care Research and Quality and the National Institutes of Health identified a paucity of validated FH tools suitable for use in primary-care or public health contexts.
In 2014, the National Colorectal Cancer Roundtable found that electronic health record systems are poorly prepared to store and interpret FH relevant to colorectal cancer (CRC) risk assessment.
The US Centers for Disease Control and Prevention–sponsored Family Healthware Trial investigated behavioral change related to the clinical use of a patient-completed electronic family history tool (FHT) that provided risk assessment for several conditions, including type 2 diabetes, heart disease, stroke, and colorectal, breast, and ovarian cancer.
The FHT was not designed for stand-alone CRC risk assessment, but extensive expertise was brought to bear in its development, suggesting that the FHT may be a useful point of comparison for the evaluation of other automated CRC risk assessment tools. The public version of the My Family Health Portrait (MFHP) tool was developed and launched in 2004 as part of the US Surgeon General’s Family History Initiative. The tool was designed to help the public collect, organize, and share FH information with relatives and health professionals.
The current version of the MFHP was designed to collect FH in a format compatible with electronic health record systems using nomenclature and data standards for storing and sharing FH; the MFHP has been studied in several settings and is freely available to the public.
Family Health History Multi-Stakeholder Workgroup of the American Health Information Community
New standards and enhanced utility for family health history information in the electronic health record: an update from the American Health Information Community’s Family Health History Multi-Stakeholder Workgroup.
An important limitation of the current public MFHP tool is that it does not provide users with feedback or education regarding their individual disease risk. Here, we report on the development of a CRC risk assessment module compatible with the public MFHP tool and, potentially, electronic health record systems. Moreover, we present data regarding the analytic validity of the new module and the FHT according to the analytical validity; clinical validity; clinical utility; and ethical, legal, and social issues (ACCE) framework proposed as a method for evaluating family health history and other genomic health tools.
The MFHP tool output is a structured XML output file that is computable for risk assessment. An algorithm based on National Comprehensive Cancer Network and United States Preventive Services Task Force CRC risk assessment guidelines was developed to dichotomize individuals into “elevated” and “not elevated” risk categories based on data entered into the MFHP tool (Supplementary Appendix S1 online). To avoid inducing potentially inappropriate distress in individuals without benefit of immediate access to evaluation by a health professional, individuals at “elevated” risk were not further subdivided into “high” and “moderate” risk categories by the MFHP tool. The model did not incorporate other data elements, such as proband gender, body mass index, or self-identified race, because there were no evidence-based guidelines for assigning risk for these factors in conjunction with FH data. The final algorithm was reviewed by a group of cancer risk assessment experts prior to being programmed into an open-source software tool. This input resulted in the final decision to provide a dichotomous rather than a more quantitative risk assessment and to consider all reported polyps as potential sources of risk warranting further discussion with a health-care provider. The output of the MFHP tool is risk-specific CRC information printable as letters for public users and their health-care providers.
Study population and risk algorithm validation
We used 150 patient-entered pedigrees sequentially derived from the ClinSeq cohort as the substrate for assessing the MFHP tool and the US Centers for Disease Control and Prevention FHT CRC module’s analytic performance.
The cohort consists of self-selected volunteers with an interest in learning their personal genomic risks through genome or exome sequencing; individuals entered the study without investigator foreknowledge of cancer FHs. The cohort is more likely to be white, well-educated, and affluent than the general US population. Forty additional patient-entered pedigrees derived from the cohort served as internal controls; 20 were deliberately modified to increase CRC risk to different levels (10 “strong” risk and 10 “moderate” risk) by adding additional accepted risk factors to the pedigree data.
Three different methods were used to derive risk estimates from the test pedigrees: the MFHP tool (including the new CRC risk module, referred to as MFHP hereafter); the Centers for Disease Control and Prevention FHT (1.0 Beta Version for Research Studies); and independent review by three genetic counselors (GCs) with expertise in CRC risk assessment. The GCs were blinded to which pedigrees had been modified to alter risk, the risk assessments made by the MFHP tool, the FHT, each other’s risk assessments, and proband self-reported body-mass index and race. The GCs were instructed that they should categorize the proband’s CRC risk as “weak,” “moderate,” or “strong” according to the criteria outlined by Yoon et al.
Risk estimates of “weak” by the FHT and GC review were considered “not elevated,” whereas “moderate” and “high” risk assignments were considered to be “elevated.” Pedigrees were scored as showing “elevated” risk for CRC only when at least two of three counselors reported risk levels as “moderate” or “strong” (true positives). The κ statistics were generated to assess GC interrater agreement for CRC assessment using the “weak,” “moderate,” and “strong” risk estimates for all 190 pedigrees. For the calculations of sensitivity, specificity, positive predictive values (PPVs), negative predictive values (NPVs), and κ statistics for the presence of CRC risk for the MFHP tool and the FHT versus the GC review, only the 150 unmodified pedigrees were used.
Results
GC risk estimates for CRC in the ClinSeq pedigrees
Sixteen of the 150 (11%) unmodified pedigrees were identified as having “elevated” risk by at least two of three GCs rating risk as “moderate” or “strong”; 6 of 150 (4%) were identified as having “strong” risk by at least two of three counselors. For the modified internal control pedigrees, the three counselors agreed for 10 of 10 (100%) and 6 of 10 (60%) pedigrees intended to be at “strong” or “moderate” risk, respectively. Pair-wise κ statistics for the three counselors calculated for all 190 pedigrees were 0.760, 0.776, and 0.851, indicating substantial agreement between the GCs according to the Landis–Koch guidelines.
Comparison of the MFHP tool versus expert GC risk estimates
Twenty-seven of the 150 (18%) unmodified pedigrees were assigned “elevated” risk status by the MFHP tool. The MFHP tool had sensitivity and specificity of 81% (95% confidence interval (CI): 54–96%) and 90% (95% CI: 83–94%) for pedigrees judged to be “elevated” by GC review. The PPV and NPV were 48% (95% CI: 29–68%) and 98% (95% CI: 93–99%), respectively (Figure 1). All 20 of the modified pedigrees intended to be at “strong” or “moderate” risk were identified as being at “elevated” risk by the MFHP tool. The κ statistic comparing the MFHP tool and GC review suggested moderate agreement (0.54).
Figure 1Diagram showing the process used for developing interpretations of 150 unmodified pedigrees derived from the ClinSeq population (see text) and resulting calculated values for sensitivity, specificity, PPV, NPV, and calculated κ for the My Family Health Portrait colorectal risk algorithm for detection of elevated colorectal cancer risk using expert genetic counselor pedigree interpretation as the “gold standard” for risk interpretation. CI, confidence interval; NPV, negative predictive value; PPV, positive predictive value; TN, true negative; TP, true positive.
Fourteen of 150 (9%) of the unmodified pedigrees were considered to be at “elevated” risk by the FHT; of these 14 pedigrees, 3 of 150 (6%) were classified as being at “strong” risk and 11 of 150 (7%) were rated to be at “moderate” risk. The FHT had sensitivity and specificity of 75% (95% CI: 48–93%) and 99% (95% CI: 95–100%), respectively, as compared with pedigrees judged to be “elevated” by GC review. The PPV and NPV for FHT were 86% (95% CI: 57–98%) and 97% (95% CI: 93–99%), respectively. The κ statistic comparing the FHT with GC review indicated substantial agreement (0.78). For the modified control pedigrees, there was agreement for 10 of 10 (100%) between the FHT and pedigrees intended to be at “strong” risk. Eight of 10 (80%) of the modified pedigrees intended to be at “moderate” risk were identified as such by the FHT; for 2 of 10 (20%) the FHT rated the risk as “strong” instead of “moderate.”
Assessment of disagreement between GC risk estimates and the MFHP
There were three unmodified pedigrees that were identified as being at “elevated” risk by GC review that were rated “not elevated” risk by the MFHP tool (“false negatives”) and 14 pedigrees for which the MFHP tool rated risk as “elevated” and GC review rated risk as “not elevated” (“false positives”). The sources of the discrepancies were diverse (Table 1). The “false-negative” pedigrees shared the characteristic of a lack of CRC cases or colorectal polyps in the proband or relatives. For one of the pedigrees, the error source was a failure to enter endometrial cancer as a structured term due to misspelling. The “false-positive” pedigrees often appeared to arise from differing assignment of risk arising from personal or family histories of colon polyps or a history of CRC or CRC-associated cancers in second-degree relatives.
Table 1Potential sources of heritable risk for CRC present in pedigrees yielding “false-negative” and “false-positive” results with the MFHP in comparison with a gold standard of pedigree review by three expert GCs
Table 1Potential sources of heritable risk for CRC present in pedigrees yielding “false-negative” and “false-positive” results with the MFHP in comparison with a gold standard of pedigree review by three expert GCs
Discussion
This study provides preliminary evidence supporting the analytic validity of an automated, consumer-oriented screening algorithm for heritable CRC risk assessment that is compatible with the US Surgeon General’s MFHP. To our knowledge, it is the first study to formally evaluate the predictive value of an FH risk algorithm designed specifically to work with the MFHP tool. It is important to note that the “gold standard” in this study is not development of disease but rather the risk prediction arrived at by a group of three expert cancer GCs. The PPV values for the MFHP and the FHT exceed the 10% threshold that is considered an acceptable value for PPV for a genomic screening test.
Because only a fraction of individuals with elevated CRC will actually develop disease, the PPV regarding development of clinical disease is likely to be lower than the values observed in this study, and the NPV is likely to be higher, for both the MFHP and FHT tools.
The MFHP algorithm was not designed to be diagnostic of CRC syndromes; rather, it was designed to be a potential component of an educational tool that helps individuals and their health-care providers stratify risk for heritable CRC in order to have an informed conversation regarding further evaluation. As a screening tool intended for use by the general public, it was designed to have a higher sensitivity at the cost of specificity, based on the rationale that missing elevated risk of colon cancer incurs a greater penalty than encouraging additional patients to discuss their risk of CRC with a health-care provider. This choice could be debated because available evidence suggests FH is not a strong predictor of absolute risk for CRC in the population.
Evaluation of the pedigrees for which the risk ratings were discordant between GC review and the MFHP tool suggests that the MFHP tool assigns “elevated” risk to pedigrees with second-degree relatives having CRC or multiple Lynch syndrome–related cancers, or a personal history or FH of colon polyps. The GCs might not attribute much risk to histories of colon polyps lacking details of pathology. The FHT also does not fully incorporate personal and FH of colon polyps when assigning risk. Assuming such polyps are benign may underestimate risk, and at least one counselor desired additional information.
Two of three instances for which the MFHP tool did not assign elevated risk represent somewhat gray areas of risk assessment for CRC. The lack of clarity in guidelines for such gray areas is reflected in the inconsistency in GC risk assignment; for two of three of the “false-negative” pedigrees, at least one expert GC rated the risk as “weak” and another rated the risk as “strong.”
This study has a number of important limitations. Measured internal agreement between the counselors suggests that expert GCs do not always concur when independently assigning CRC risk to pedigrees. The discrepancies between risk estimates provided by GC review and the MFHP algorithm highlight that, currently, there is no universally accepted “gold standard” for the detection of CRC risk in FHs in routine clinical care. Similar to the 2005 study by Qureshi et al.
, this study uses review by expert GCs to arrive at a proxy gold standard for CRC risk. Additional limitations of this study include the small sample size, the nature of the ClinSeq population (which may not represent the full diversity of the US population), and that the study was only able to examine the analytic validity (and not the clinical utility) of the MFHP CRC algorithm. Future studies should seek to replicate these findings and define the clinical validity and clinical utility of the MFHP tool in additional patient populations.
Disclosure
The authors declare no conflict of interest.
Acknowledgements
This project was supported by the Genetic Disease Research Branch and the Genomic Healthcare Branch, National Human Genome Research Institute, National Institutes of Health. The authors thank the following individuals for contributions made to developing the risk assessment algorithm and for their thoughtful comments on the manuscript: Andrew Freedman, Mitchell Gail, Donald Hadley, David Lanier, Colleen McBride, Gurvaneet Randhawa, Maren Scheuner, Rodolfo Valdez, and Daniel Wattendorf. Some of the data in this manuscript were presented in abstract form at the 2011 meeting of the American Society of Human Genetics in Montreal, Canada.
Family Health History Multi-Stakeholder Workgroup of the American Health Information Community
New standards and enhanced utility for family health history information in the electronic health record: an update from the American Health Information Community’s Family Health History Multi-Stakeholder Workgroup.
03207-X&id=gr1.jpg){kind=link}