Familiarization with a New Technology for the Laboratory Before Validation
The laboratory with little or no experience with microarray technology should become familiar with all aspects of the new technology before beginning the validation process, regardless of the regulatory status of the array. Familiarization begins with understanding the processes, features, and capabilities of the technology selected. The laboratory should gain experience with the instrumentation, platform design, software, reagents, methodology, technological limitations, workflows, DNA quality parameters, etc., by experimental sample runs. Similarly, the laboratory should become familiar with the features of each sample type the laboratory will process, as different sample types may have unique considerations for microarray data quality and clinical applicability. The laboratory must also be familiar with the potential imbalances and rearrangements associated with the clinical indications.
The use of samples well characterized as “normal” and “abnormal” by another method is valuable during the familiarization process to gain experience in the recognition of CNVs that may represent true biological variation or a probe/platform performance issue. It is suggested that laboratories use a combination of data from well-characterized cases processed and run on their platform(s), data from other laboratories, and/or data available from online databases to gain and broaden their experience. Data sharing should involve a spectrum of array results and data quality.
Laboratories need to be able to recognize nonperforming (or nonresponsive) probes, technically induced artifacts, and other issues affecting data quality. Laboratories should become familiar with CNVs that are benign and/or common and resources to aid in the recognition and interpretation of CNVs.
3.- Working Group of the American College of Medical Genetics
American College of Medical Genetics recommendations for the design and performance expectations for clinical genomic copy number microarrays intended for use in the postnatal setting for detection of constitutional abnormalities..
,4.- Lee C.
- Iafrate A.J.
- Brothman A.R.
Copy number variations and clinical cytogenetic diagnosis of constitutional disorders..
,5.- Firth H.V.
- Richards S.M.
- Bevan A.P.
DECIPHER: Database of Chromosomal Imbalance and Phenotype in Humans Using Ensembl Resources..
,6.- Riggs E.R.
- Church D.M.
- Hanson K.
Towards an evidence-based process for the clinical interpretation of copy number variation..
,7.- Sayers E.W.
- Barrett T.
- Benson D.A.
Database resources of the National Center for Biotechnology Information..
,8.- Zhang J.
- Feuk L.
- Duggan G.E.
- Khaja R.
- Scherer S.W.
Development of bioinformatics resources for display and analysis of copy number and other structural variants in the human genome..
Verification and Validation
Verification of a Food and Drug Administration–approved/cleared platform
At the time of the publication of these guidelines, there are no commercially available Food and Drug Administration (FDA)-approved or FDA-cleared microarrays for this application. However, laboratories are advised to keep abreast of new developments in this rapidly developing technology.
For any FDA-approved or FDA-cleared microarrays where the laboratory plans to claim the test as FDA-approved/cleared, the approved protocol and intended use (usually included in the package insert) must be followed. The laboratory must verify that it can obtain comparable performance specifications as those established by the manufacturer with regard to accuracy, precision, and reportable range of results.
At the onset of verification, pass/fail criteria for the verification protocol should be established. If the prespecified acceptance criteria are not met, and a repeat or evaluation of the reasons for the failure does not resolve the concern, the laboratory should consider whether or not the array is appropriate for clinical testing.
Accuracy testing will measure the ability of the platform and software to detect known abnormalities. The accuracy evaluation is accomplished by running a series of previously characterized abnormal cases (this may be accomplished through sharing samples with an established laboratory). A minimum of 15 cases is recommended. To the extent possible, the laboratory should use abnormal samples that represent abnormalities that the array is designed to detect. This evaluation should include both a comparison of the findings from the region(s) expected to be abnormal as well as a comparison of the rest of the genome analyzed by the platform. The laboratory must document the concordance of the expected results and any unexpected findings. Because this technology may detect true alterations not previously identified, any unexpected findings that fall within the determined reportable range (as defined in “Validation of a new platform for the laboratory” section) should be further investigated to determine whether the finding represents true biological variation. This will involve the use of an alternative technology or microarray platform for correlation of the unexpected finding.
Precision testing should measure the reproducibility of repeated tests for the same result. The precision of the platform is established by running a minimum of two abnormal samples, each run multiple times in separate experiments. The concordance of the repeated runs should be documented, and any alterations should be considered (variability of breakpoints, calls, and potential reasons for variation, i.e., segmental duplication-rich region) as they pertain to the reportable range, functional resolution, and potential variability around breakpoints. Some variability around breakpoints may be expected due to segmental duplications and individual probe performance. Precision testing can allow for an assessment of breakpoints and potential impact on the clinical interpretation. Breakpoint variability that does not alter the clinical interpretation would be less concerning than variability that does alter the interpretation. Samples with multiple abnormalities are preferable as they maximize the number of findings for the precision study.
Any modification to the FDA-approved use of the product (as specified in the package insert) will be considered as off-label use, and therefore the microarray must then be validated as a non-FDA-approved platform.
Validation of a non-FDA-approved platform
All platforms intended for clinical testing must be either FDA-approved/cleared and verified or must be validated by the performing laboratory. Validation is the process by which the laboratory measures the efficacy of the test in question by determining its performance characteristics when used as intended. This is necessary to demonstrate that it performs as expected and achieves the intended result. Validation is required when using laboratory-developed tests or modified FDA tests. The method and scope of the validation must be documented.
At the onset of validation, pass/fail criteria for the validation protocol should be established. If the prespecified acceptance criteria were not met, and a repeat or evaluation of the reasons for the failure does not resolve the concern, the laboratory should consider whether or not the array is appropriate for clinical testing.
The extent of work necessary for a validation can depend in part on whether the laboratory is validating a new microarray platform for the laboratory, validating a modified design of a previously validated platform, or adding additional sample types or intended uses to a previously validated platform. A new platform is defined as any new methodology or array introduced to the laboratory. A single microarray vendor may produce multiple similar platforms, but each must be validated independently. A modified design may include either minor modification to probe coverage, either through manufacturing of the array or by in silico probe filtering.
Validation of a new microarray platform for the laboratory
Through the validation process, the laboratory must establish the performance characteristics of the microarray platform and accompanying software. The performance characteristics that must be established include the accuracy and precision of results, the analytical sensitivity and specificity, and the reportable ranges. Validations should be documented for each array platform used for clinical testing, regardless of whether the laboratory has prior experience with a different platform.
The reportable range of results includes criteria to identify a CNV and criteria to report a CNV. Laboratories, with consideration of the manufacturer’s recommendations, should identify the parameters specific to their platform (number of consecutive probes, log2 ratios, SNP allele ratios, QC metrics, etc.) that are necessary to conclude that a copy number call represents a true CNV. As the functional resolution is a combination of probe density and number of probes necessary to identify a true CNV, the reportable range should be at or above the functional resolution of the platform. The reportable range should be determined before the evaluation of the validation set, and data from the familiarization process should be utilized. The reportable range may exclude well-characterized benign CNVs. If the reportable range is altered for the laboratory, the validation data should be re-evaluated with the new reportable range. However, if the previously identified validation samples do not contain abnormalities that challenge the altered reportable range, additional samples should be evaluated.
The accuracy evaluation is accomplished by running a minimum of 30 previously characterized abnormal controls. To the extent possible, the laboratory should use abnormal controls that represent abnormalities that the array is designed to detect. This should include both autosomal and sex chromosome abnormalities as duplications and deletions on the sex chromosomes may behave differently in each sex. Furthermore, blinding the evaluators to the expected abnormalities has the additional benefit of validating the settings, evaluation of data, and reportable range. Samples used for validation should represent a variety of findings with various sizes of abnormalities, combinations of gains and losses, various regions of the genome, and some aberrations that challenge the technical limits of detection for reportable DNA gains and losses.
Sample exchanges with a laboratory that is proficient with a similar microarray platform can provide a good source of samples for validation. Exchange of validated data sets (e.g., array files) between laboratories is recommended for additional experience in data analysis.
This evaluation should initially include a full review of the data to identify aberrations that meet the reportable range while blinded to the expected abnormality (as would fit the clinical workflow), followed by an unblinded comparison of the findings from the region(s) expected to be abnormal, as well as an evaluation of the rest of the genome analyzed by the platform. An evaluation of the regions expected to be normal is also important in assessing the probe behavior across the genome. The laboratory must document the concordance of the expected results and any unexpected findings. Evaluation should also include breakpoint evaluation with regard to gene content and genomic architecture. The laboratory should also recognize nonresponsive probes in a region expected to show loss or gain (this may be due to either poor performance probes or underlying genomic architecture). As this technology may detect true alterations not previously identified, any unexpected CNVs that fall within your laboratory-determined reportable range should be further investigated to determine whether the finding represents true biological variation. This may involve the use of an alternative technology or microarray platform for correlation of the unexpected finding. As both expected and unexpected findings are evaluated, careful selection of the 30 samples is important and the ability to evaluate unexpected findings in the 30 samples should be considered.
Sensitivity and specificity are determined by the number of true-positive, true-negative, false-positive, and false-negative results in a validation data set that meet reporting criteria. However, for a whole-genome assay, all true positives and true negatives are not known. Therefore, specificity and sensitivity for genome-wide array platforms cannot be calculated as traditionally defined.
Sensitivity is evaluated by comparison of expected versus observed abnormalities, and this is then extrapolated to the rest of the genome. Rather than a traditional calculation of specificity, an evaluation of the positive predictive value of the assay is desirable. Determination of the positive predictive value will involve the identification of copy number calls that fall within the laboratory’s determined reportable range and a determination of the proportion of those calls that are true. To improve the specificity of the platform, if certain probes are recognized to repeatedly act as false positives, these probes should be removed from future analyses. The identification of false-positive probes may be due to technical or biological variables considering that not all regions of the genome are amenable to accurate locus-specific evaluation of copy number with this technology. If probe content is masked by the laboratory, these changes should be documented. If the changes are sufficient to alter the performance of the platform, an evaluation of the validation data with the altered probe content is required.
The precision testing should measure the closeness of repeated test results to one another. The precision of the platform is established by running a minimum of two abnormal samples, each run multiple times in separate experiments. The concordance of the repeated runs should be documented, and any alterations should be considered (variability of breakpoints, calls, and potential reasons for variation, i.e., segmental duplication-rich region) as they pertain to the reportable range, functional resolution, and potential variability around breakpoints. Some variability around breakpoints may be expected due to genomic architecture and individual probe performance. The precision testing can allow for an assessment of breakpoints and potential impact on the clinical interpretation. Breakpoint variability that does not alter the clinical interpretation would be less concerning than variability that does alter the interpretation. Samples with multiple abnormalities are preferable as they maximize the number of findings for the precision study.
Validation of a new version of a previously established platform
The definition of a new version should be limited to those situations in which a minimal number of probes are removed, added, and/or replaced for the purpose of improved performance, and/or coverage is enhanced over a limited number of genomic regions. This would likely involve <10% of the total probe coverage, with no more than 5% probe removal. It should be recognized that these types of changes to an established platform are likely a rare event and most changes in platforms will require a full validation.
In the laboratory that is proficient with microarray technologies, a new version of a platform in use by the laboratory from the same manufacturer should be validated with a minimum of five abnormal samples.
Known abnormal samples from the previous version should be run using the new platform version for comparison to ensure that the performance meets the laboratory standards and to assess performance of probes added into higher-resolution platforms.
New content on an upgraded version should be assessed, if possible, using known abnormal sample(s) with variation in the region of the new content to determine performance.
The evaluation of this validation set of at least five samples should include data analyzed to determine whether the platform and software detected the expected abnormality. If other abnormalities are detected that meet the laboratory-reporting range, the validation should determine whether the findings represent true biological variation.
Validation of additional sample types on an established platform
It is expected that the initial validation will involve the most common sample type for the expected intended use. For example, if the intended use is postnatal evaluation, the sample type may be DNA extracted from peripheral blood. Because the quality of the DNA may vary from alternative tissue sources and this may add interference factors to the microarray analysis, use of DNA from alternative sample types requires an evaluation of the potential for interference.
9.- van Heesch S.
- Mokry M.
- Boskova V.
Systematic biases in DNA copy number originate from isolation procedures..
For a new sample type, an evaluation of the impact of the new sample type on data quality is necessary. The DNA extraction process should be part of the validation process. If there will be minimal changes to the processing or analysis, then a validation of the new sample type can involve equivalency of data quality with the new sample type. If alterations are made in the processing of the array or analysis (e.g., change of reference set), then a new validation is required. In addition, if the new sample type requires a different reportable range, then a new validation is required.
Validation of the allelic differentiation potential of SNP-detecting platforms
The detection of AOH is not in and of itself diagnostic but can identify a concern that would require additional testing such as sequence-based mutation analysis or uniparental disomy testing. However, as AOH may be reported by the laboratory, evaluation of the performance of the SNP-detecting probes to define regions of AOH should be included in the validation. Reports of these findings must clearly state that the finding is not diagnostic.
Given sufficient probe density, there should be a correlation between copy number state and SNP allele state.
A minimum of five samples need to contain expected copy neutral AOH in addition to CNVs. Interlaboratory comparisons of samples that contain known uniparental isodisomy or regions identical by descent are recommended. This comparison should address the data types that would be included in a report, such as approximate regions of AOH and approximate percentage of genome identical by descent. The detection and accurate size assessment of AOH by SNP-based arrays depend on the density of SNP probes. If the validation method does not address accuracy of breakpoints in AOH calls, reports should reflect this uncertainty. Inaccurate size estimation for regions of AOH could lead to unwarranted follow-up testing for uniparental isodisomy, somatic loss of heterozygosity, and/or autozygosity mapping.
10.- Mason-Suares H.
- Kim W.
- Grimmett L.
Density matters: comparison of array platforms for detection of copy-number variation and copy-neutral abnormalities..
Mosaicism detection
Low-level mosaicism for unbalanced rearrangements and aneuploidy may not be detected by microarray analysis. In addition, the level of detectable mosaicism will vary by size, region of genome, copy number state, DNA quality and data quality. Therefore, it is not likely that a specific level of mosaicism can always be identified uniformly throughout the genome, and this limitation should be recognized.
Without extensive validation to determine specific levels of mosaic detection for a wide variety of CNV sizes and genomic regions, it is not recommended that this technology be used to rule out mosaicism. However, experience in mosaicism identification is desirable to maximize opportunity for detection. Methods for determining detectable levels of mosaicism include dilution studies and analysis of the sample by other quantitative methods. Fluorescence in situ hybridization analysis of fresh (uncultured) samples provides a reliable means to establish the level(s) of mosaicism detectable by microarray. Conventional cytogenetic analysis of metaphase cells provides information about mosaicism but may not accurately reflect levels of mosaicism. The laboratory director should determine the method used by the laboratory. More than one method is recommended.
For cells in suspension, dilution studies using samples with known CNVs may help to determine detectable levels of mosaicism. This method can provide an effective means to establish thresholds; however, it may have limitations as an artificial method. Dilution studies for SNP-detecting arrays may not be possible because they may introduce additional genotypes that complicate the analysis.
The detection of mosaicism may include information from both the log2 ratio and the SNP allele pattern as applicable for each platform.
Be aware that microarray analysis gives a relative level of copy number across the cells within the sample but does not provide a cell-by-cell determination of copy number (e.g., trisomy in 60% vs. tetrasomy in 30% of cells).
Special considerations for validation of prenatal specimens
Experience with postnatal arrays and with common and rare CNVs is important for the processing and interpretation of array results for prenatal specimens. For validation, a distinction should be made between cultured amniocytes and chorionic villus sampling (CVS) cells and uncultured (direct) amniocytes and villi. The validation performed depends on whether the platform has been previously validated for postnatal use or is new to the laboratory and whether both cultured and uncultured cells will be used.
For cultured amniocytes and CVS cells, if prenatal array analysis is performed on an array platform new to the laboratory, the issues and process discussed in the “Validation of a new platform for the laboratory” section apply, and a minimum of 30 previously characterized cases should be processed. Due to the difficulty of obtaining abnormal prenatal specimens, this collection of 30 samples will likely include some previously characterized as normal cases. Therefore, additional experience with abnormal array findings through additional tissue types and data exchanges should occur, to ensure that a wide variety of abnormalities have been evaluated both in-house and in silico.
For a previously validated platform for postnatal use, the addition of prenatal specimens requires an understanding of the potential issues that these samples can present regarding data quality. The DNA extraction process should be part of the validation process. If the laboratory will perform analysis on cultured amniocytes and CVS, both sample types should be represented in the validation.
Prenatal samples (including products of conception).
Healthy cultures established from amniocytes, villi, and fetal tissue yield an adequate quantity and quality of DNA and can be viewed as essentially equivalent for validation purposes. However, the laboratory should be aware of factors that can affect DNA yield and data quality including culture age, growth rate, confluency, and shipping conditions.
Because uncultured cells may yield different amounts and quality of DNA, additional validation is required to become familiar with potential differences as compared with cultured cells. Parameters to consider for uncultured amniocytes include method of DNA extraction, volume, and gestational age given that these parameters influence the amount and quality of DNA. For example, uncultured amniocytes yield less DNA than cultured cells; however, the quality of the DNA is generally higher from uncultured cells.
Because villi represent a more complicated tissue with different cell types/layers (syncytiotrophoblast, cytotrophoblast, and mesenchymal core), DNA may be extracted from all cell types, or the laboratory may eliminate or concentrate on different cell layers for DNA extraction.
11.Cytogenetics TAfC. Professional Guidelines for Clinical Cytogenetics and Clinical Molecular Genetics: QF-PCR for the Diagnosis of Aneuploidy, Best Practice Guidelines. In: Genetics BSfH (ed), v3.01. Clinical Genetics Unit, Birmingham Women’s Hospital: Birmingham, England, B 15 2TG: ACC Heads of Department; 2012:1–16.
Special quality assurance requirements for prenatal specimens.
Back-up cultures of all prenatal samples undergoing array analysis should be established and maintained for the purposes of (i) possible array failures on direct extractions, (ii) evaluation of possible mosaicism on an independent culture, and (iii) the need to perform metaphase chromosome or fluorescence in situ hybridization analysis to investigate CNVs.
Maternal cell contamination (MCC) analysis should be performed on all prenatal samples, unless contamination is otherwise excluded. MCC can result from direct samples of amniocytes with blood admixture, CVS samples not successfully cleaned of maternal decidua (a more frequent problem with products of conception), and cell cultures undergoing extensive subculturing resulting in expansion of maternal cells. When undetected, MCC can result in missed detection or misinterpretation of copy number changes, even in the context of a male result. Laboratories should understand that MCC can be detected by array software (i.e., SNP-based platforms) or, in the case of male fetuses, by a shift of the sex chromosome plots (mimicking mosaicism). Laboratories should also understand how the presence of MCC can affect detection of CNVs, including different types (gains and losses) and different sizes (small versus large gains and losses).
Mosaicism may be detected in prenatal samples and may represent culture artifact (pseudomosaicism), true fetal mosaicism, or, for CVS, confined placental mosaicism. Careful investigation may be required to determine the fetal genotype. For traditional chromosome analysis, algorithms have been developed to deal with confined placental mosaicism and pseudomosaicism. These algorithms will also need to be developed for microarray analysis and will depend on whether the analysis used direct or cultured cells, and if the mosaicism can be confirmed on an independent culture.
Software Considerations
The laboratory should recognize software limitations and the need for manual and visual inspection of the data for aberration and mosaicism detection because the software may not flag all relevant calls that may be identifiable by a visual inspection of the data. To verify that the method for result generation (including software calls and manual inspection) detects known aberrations at certain mosaicism levels, the system should be challenged with different types of aberrations. During the familiarization phase, the settings should be explored and optimized for aberration detection and then established parameters should be used consistently throughout the validation process. Algorithmic parameter settings may be different for various sample types.
Changes to the software settings from those used during the validation may require a re-analysis of at least a subset of the validation data using the new settings to identify any changes to the performance characteristics of the microarray platform. Such changes may include, but are not limited to, new annotation libraries, changes to any in silico reference set, or any changes to the aberration-calling algorithm.
The laboratory should understand that most normalization algorithms assume a primarily diploid state, which may obscure the detection of polyploidy. In some situations, the allele pattern may assist with the detection of triploidy but may not identify tetraploidy. Both situations are likely rare in the postnatal constitutional population but may be present as mosaic findings as well as in prenatal settings.
The laboratory should document the software, parameters, and rules used in the analysis of the microarray, as well as all limitations of the analysis program.
Quality Control
Identification
For each array, the slide identification number, sample sex, control sex, and sample-tracking control (if applicable) should be verified. Discrepancies in the documentation from the physical sample should be investigated and resolved before processing.
DNA requirements
The laboratory should establish the minimum DNA requirements to perform testing. Each laboratory should have established parameters for the determination of the sample quality and quantity and criteria for adequacy of each. If a sample does not meet these minimum requirements and is deemed suboptimal, the recommended action is to reject the specimen and request a repeat specimen.
Equipment calibration, maintenance, and QC
Equipment, instrumentation, and methodologies employed during the validation and use of microarray platforms should be calibrated, monitored for QC, and regularly maintained as appropriate. Quality metrics should be established whenever possible throughout the assay. Laboratories should ensure that documentation and safeguards are provided by the software manufacturer and that data are processed and summarized in a consistent fashion for every clinical analysis. Most analysis software provides a hierarchy of users with customizable permissions, which enable the laboratory director or supervisor to prevent modification of analysis settings so that all specimens are analyzed consistently. Any changes to data processing should be validated and documented.
General QC metrics
Every microarray platform has defined quality metric values, e.g., adequate dye incorporation and/or amplification, fluorescence intensities, signal-to-background noise ratio, and standard deviation or standard error. Standard cutoff values and acceptable limits should be established for these metrics to ensure that the generated results are reliable and precise enough to be used for a clinical assessment. Quality metrics should be monitored for DNA labeling, hybridization efficiency, data generation and analysis, and other platform-specific parameters. The QC metrics should be incorporated into the laboratory quality assurance and quality improvement programs to monitor analytical variables.
Data quality
The quality of the data will affect the ability to detect genomic aberrations. Therefore, it is absolutely necessary for the laboratory to understand the within-array metrics provided by the analysis software and how each of these metrics reflects the quality of the data. There are a number of metrics that provide a measurement of signal to noise (i.e., artifactual random variance unrelated to genomic location) in the data, such as the difference between the log ratio values of consecutive probes and the spread of the derivative log ratio values after outlier rejection. Similar metrics of variance exist for each platform.
The laboratory should establish acceptable ranges for each QC metric chosen to represent data quality by the laboratory. These ranges are often provided by the manufacturer. However, the laboratory may want to modify these ranges on the basis of their experience with the arrays during the validation process. The laboratory should establish criteria for next steps, should the data fall outside of these established ranges.
Annotation/databases
An integral part of the data analysis is accessibility and use of private and public annotations/databases during the analysis process. Because these annotations are critical for interpretation, it is important that these tools are carefully constructed and applied by the laboratory or software manufacturer. All critical annotations should be thoroughly vetted, and the source(s) should be verified. For all reportable calls, the genomic content should be verified by an independent database source. The manufacturers should provide mechanism(s) for updates to these annotations. Documentation of resources and databases accessed for interpretation is recommended.
Verification of new lots of microarrays/reagents
Verification should ensure that new lots of microarray slides and/or reagents perform in the same manner as the previous lot. The manufacturer should supply documentation of the QC comparison between lots (e.g., oligonucleotide synthesis verification, accuracy of SNP calls or other defined control parameters). New lots of reagents (e.g., new labeling kits, consumables) should have documented equivalency between runs. This may be accomplished by documenting that the QC metrics meet certain set parameters for the new lot of reagents.
Confirmation of specific CNVs
With proper technical performance and analytical validation, it should not be necessary for the performing laboratory to further confirm a CNV called with the laboratory-validated parameters, after the validation stage. Each laboratory should establish a threshold (number of probes and/or genomic size, as well as other QC metrics) for declaring what constitutes a reportable abnormality with their assay. Features to keep in mind when assessing copy number changes are the appropriate log ratio difference between data, the presence of uniform contiguous probe behavior within and adjacent to call, sharp copy number state transitions at breakpoint boundaries, supportive SNP allele states (when applicable), and evaluation of least processed log2 ratio data (e.g., weighted versus not weighted). Any call-specific quality score provided by the software may be considered.
Given that it is desirable to maximize detection of aberrations of clinically important genes and of aberrations in mosaic form (which may not generate a robust copy number call), it is acceptable and appropriate at the discretion of the performing laboratory to evaluate calls that do not meet the laboratory-validated parameters. These calls may be flagged for review and correlated with the patient’s clinical indication, but should be confirmed by an independent methodology if reported.