REGULATIONS & STANDARDS
 |
Jan S. Krouwer, PhD, is president of Krouwer Consulting (Sherborn, MA), which provides statistical and reliability consulting to the IVD industry. He can be reached at jan.krouwer@comcast.net.
|
Waiver assays are “simple laboratory examinations and procedures that are cleared by FDA for home use; employ methodologies that are so simple and accurate as to render the likelihood of erroneous results negligible; or pose no reasonable risk of harm to the patient if the test is performed incorrectly.”2 The advantage of achieving a waiver classification for assays is that labs with a waiver certificate can perform such assays. Since 63% of labs have a waiver certificate, an assay with waiver status can be sold to many more labs.
The key difference between the proposed and previous waiver guidances is the assessment of performance. Historically, most assay evaluations obtain performance information by estimating one of the following:
- Average bias and imprecision.
- 95% of the differences between the assay results and a comparison method.3
- 95% uncertainty interval for the results.4
While FDA has traditionally used the first method, the second method is appearing more often in the literature. However, for the second and third methods, performance goals are often lacking. Although much has been written about goals for medically acceptable differences between assay results and reference methods, there are few standards that address this issue.5 One standard that has medical acceptability limits is ISO 15197 for home-use glucose assays. This standard states that in order to achieve medically acceptable performance, 95% of the results should be within stated limits.6 But if this goal were met, then for every 1 million results, up to 50,000 medically unacceptable results would be allowed.
 |
Figure 1. Total error and outliers.
|
What is missing in ISO 15197 is presented in Figure 1, which adds limits for outliers. Outlier is a generic term for a large error. Outliers can be classified as either a statistical outlier, which fulfills the criteria of a specific test such as being greater than 3 standard deviations from the mean, or an erroneous result, which exceeds a stated limit and may or may not be a statistical outlier. The proposed waiver guidance document focuses on erroneous results.
Based on ISO 15197, 95% of test results should be within region A in Figure 1. Values that are just outside of region A would probably not cause problems, which is shown as region B. However, results in region C will likely cause patient harm. In addition to requiring that most (95%) of the values fall in region A, very few (ideally zero) results should be in region C.
IVD manufacturers are required by the waiver guidance to implement risk management tools (e.g., hazard analysis, failures mode and effects analysis (FMEA), fault trees) to ensure that severe test errors do not occur.7 For example, hazard analysis looks for ways in which severe errors are identified and mitigations are established to prevent such errors from occurring. While risk management is not really new, considering the design control requirements for hazard analysis, the proposed waiver guidance includes an additional emphasis that reinforces the importance of hazard analysis to ensure that dangerous test results are not released.
What is new in the guideline is a different analysis for a method comparison study. Risk management differs from method comparison studies in several ways. For example, a hazard analysis is a model that attempts to predict how a system can fail. The problem with any model is that there is no guarantee it is correct. A method comparison is an empirical study that simply collects results. A big difference from the reference in a method comparison study may be observed regardless of whether there is an explanation for it or not.
Method Comparison Study
Protocol. The protocol for a method comparison study requires 360 samples to be tested by both the candidate assay, or waiver method, and the reference or comparison method. The proposed CLIA waiver guidance permits the following three types of comparison methods:
- Standard reference methods.
- Comparison methods that are traceable to a reference method. Traceability means that “the results of measurement can be related to a stated reference method, usually a national or international standard, through an unbroken chain of calibrations of a measuring system or comparisons where measurement uncertainties have been documented at every step in the procedure.”1,8 Traceability can partly be achieved mathematically from regression studies, which generate a slope of one and intercept of zero.
- Comparison methods that are similar to those stated above, except that the slope of one and intercept of zero are not achieved. If such comparison methods are used, it is suggested that IVD manufacturers should contact FDA.
The samples must be patient samples instead of controls. Unfortunately, the waiver guidance allows only 60 samples to be spiked, while at the same time requiring the samples to span the measurement range and represent equally low, medium, and high concentrations. If this number of spiked samples is not adequate, the guidance suggests contacting FDA. In addition, the intended lab operators of the tests must conduct the method comparison study. This change is an improvement over past guidances, which required operators with no previous laboratory experience.
Analysis. This review focuses on applying quantitative methods for the cases in which the comparison method is either a standard reference method or a comparison method traceable to a reference method.
Regression. Regression analysis is relegated to coming after calculating descriptive statistics. Rather than being the primary analysis method, regression is part of the background.
Total Error and Erroneous-Results Analysis. This is the main change between the proposed and previous waiver guidance, and follows recommendations and the standard EP21A by the Clinical Laboratory and Standards Institute (CLSI; Wayne, PA).9,10 The proposed guidance is requiring that regions A and C in Figure 1 be quantified. In the guidance, FDA defined region A as the allowable total error (ATE) zone and regions C as the limits of erroneous results (LER) zones.
 |
Figure 2. Total error and outliers.
|
FDA goes one step further than Figure 1 by requiring a Parkes glucose error grid for the assay that is being evaluated (see Figure 2).11 This grid makes the LER zones specific to those cases that are likely to cause patient harm.
In Figure 2, at least 95% of test results should fall in region A, and no results should fall in region C. This implies that up to 5% of results could be in region B. The proposed waiver guidance could be clearer by defining region B, which is unnamed in the document. The guidance is also confusing since it discusses results within the ATE or LER zones; but from the context, the guidance actually means results within the ATE zone and up to, but not in, the LER zones.
What Does This Mean for IVD Manufacturers?
 |
Table I. Key differences between the previous and proposed CLIA waiver guidances.
|
Since most IVD manufacturers rigorously pursue the investigation of outliers during product development, the goals of having 95% of test results in region A and 0% in region C should be met. Table I lists the main differences between the previous and proposed CLIA waiver guidances.
For many assays, IVD manufacturers will have to construct an error grid in order to calculate ATE and LER. There is a clear benefit to constructing such an error grid. For example, if a glucose test result is 160 mg/dl and the reference value is 40 mg/dl, this erroneous difference of 120 mg/dl falls in the most severe error region in an error grid.12 However, this same 120 mg/dl error for another test (glucose result is 320 mg/dl, reference value is 200 mg/dl) falls in region B in an error grid. This result means that large errors by themselves can be tolerated as long as they are not in the LER zones.
From a chemistry standpoint one may wonder, did the second case happen by chance, meaning that on other occasions, could it turn out like the first case? Although there is more leeway for reference and comparison methods, they should provide unbiased and precise results. The error analysis measures differences, and the interpretation is that any differences are due to errors in the waiver method.
Such studies do not guarantee that an assay is completely free from erroneous results. For example, if there are no values beyond the LER zones in 360 samples, the estimated erroneous-result rate is 0%. However, the upper 95% confidence bound for this rate is 1%. (The guidance asks for confidence intervals to be calculated.) Therefore, IVD manufacturers can guarantee no more than 10,000 erroneous results (i.e., results that can cause patient harm) per 1 million reported results. This is not necessarily a problem with the waiver guidance, but rather a reflection of the fact that it is difficult to prove that rare events do not happen.
There is a problem with the waiver guidance with respect to the ATE zone. For assays that have CLIA limits, the guidance requires 95% of the test results to meet such limits. This differs from current CLIA goals that require only 80% of results to meet CLIA limits.13 To be consistent with the CLIA goals, the waiver guidance should expand the CLIA limits when requiring 95% of results to meet such limits.
Process Capability
While process capability statistics (e.g., Cpm) are largely unknown for IVD assays, the process capability concept is relevant to the proposed CLIA waiver guidance.14 For example, in Figure 1, if a distribution of test differences did not go beyond region B, this would be considered a capable process and would meet the waiver guidance requirements. However, if the distribution extended into region C, then the process would not be considered capable. A process that is not capable may have no statistical outliers and could also have no failed quality control results.
Traditional Evaluations
In traditional evaluations, the method comparison results are analyzed by regression analysis using a method such as the CLSI standard EP9A2. An imprecision study is also carried out by using the CLSI standard EP5A2. In both evaluations, the results would be tested for statistical outliers, and if outliers were found, they would be discarded. The results are expressed in terms of parameter estimates such as regression slopes and intercepts, average bias, and within-run, and longer-term imprecision. One of the problems with this type of evaluation is that severe erroneous results can be missed, which are precisely the results that can harm patients.15
The method comparison analysis in the proposed CLIA waiver guidance is easier. This analysis counts the number of results falling into the ATE and LER zones. The percentages based on the results, particularly the LER percentage, provide an estimate of the risk of the waiver assay.
Risk Management
The burden of having no results in the LER zones raises the importance of risk management. Risk management involves several tools, including FMEA, which can prevent potential errors, and failure review and corrective action system (FRACAS), which can prevent the recurrence of observed errors. Both tools can be aided by fault trees and process flowcharts. Perhaps the most important tool will be FRACAS, in which the assay is repeatedly tested under as close to actual use conditions as possible to expose as many problems as possible.16 As such problems are exposed, fail-safe systems will be put in place to prevent them.
In addition, flex studies are required, which are experiments to show what happens when a range of values are tested, including those exceeding the minimum and maximum allowed. This type of risk management relies less on a model but rather on exercising the system to expose problems, which are then corrected. The actual hazard analysis submitted would document the hazards and mitigations. As with any FDA study, IVD manufacturers should practice the intended protocol to ensure that goals are achieved, which is another benefit of conducting FRACAS during product development.
Regulator’s Dilemma
Regardless of the results of the CLIA waiver protocol, regulators have to balance their decision to approve or reject an application based on not only the possible errors that might occur with the waiver assay (i.e., risk) but also its benefits. IVD assays provide valuable information, which if unavailable could increase morbidity and mortality. A waiver assay could increase the availability of an assay by having a lower selling price and by being performed in more labs. This availability would translate into more people being tested and a potentially lower morbidity and mortality.
Conclusion
The proposed CLIA waiver guidance has parts that are easier (e.g., operator and reference method requirements) and others that are more difficult (e.g., patient samples instead of controls) for IVD manufacturers. The analysis is also both simpler and more relevant. It remains to be seen whether this guidance will be the basis of a revision to the 510(k) or premarket approval (PMA) guidance.
References
01. “Draft Guidance for Industry and FDA Staff: Recommendations for Clinical Laboratory Improvement Amendments of 1988 (CLIA) Waiver Applications,” Center for Devices and Radiological Health Web site (Rockville, MD: 2005 [cited 29 September 2006]); available from Internet: www.fda.gov/cdrh/oivd/guidance/1171.pdf.
02. “Information on CLIA Waivers,” Center for Devices and Radiological Health Web site (Rockville, MD: 2001 [cited 29 September 2006]); available from Internet: www. fda.gov/cdrh/clia/cliawaived.html.
03. JM Bland and DG Altman, “Statistical Agreement for Assessing Agreement between Two Methods for Clinical Measurement,” Lancet, no. 1 (1986): 307–310.
04. ISO 101, “Guide to the Expression of Uncertainty in Measurement” (Geneva: International Organization for Standardization).
05. CG Fraser, Biological Variation: From Principles to Practice (Washington, DC: AACC Press, 2001).
06. “Requirements for In Vitro Blood Glucose Monitoring Systems for Self-Testing in Managing Diabetes Mellitus,” ISO 15197 (Geneva: International Organization for Standardization).
07. “Medical Devices: Application of Risk Management to Medical Devices,” ISO 14971:2000 (Geneva: International Organization for Standardization).
08. ISO 17511:2003, “In Vitro Diagnostic Medical Devices: Measurement of Quantities in Biological Samples; Metrological Traceability of Values Assigned to Calibrators and Control Materials” (Geneva: International Organization for Standardization).
09. JS Krouwer, “Setting Performance Goals and Evaluating Total Analytical Error for Diagnostic Assays,” Clinical Chemistry, no. 48 (2002): 919–927.
10. CLSI EP21A, “Estimation of Total Analytical Error for Clinical Laboratory Methods” (Wayne, PA: Clinical Laboratory and Standards Institute).
11. WL Clarke et al., “Evaluating Clinical Accuracy of Systems for Self-Monitoring of Blood Glucose,” Diabetes Care, no. 10 (2002): 622–628.
12. JL Parkes et al., “A New Consensus Error Grid to Evaluate the Clinical Significance of Inaccuracies in the Measurement of Blood Glucose,” Diabetes Care, no. 23 (2000): 1143–1148.
13. Code of Federal Regulations, 42 CFR 493.
14. JS Krouwer, Development and Evaluation: A Manufacturer’s Perspective (Washington, DC: AACC Press, 2002).
15. S Rotmensch and LA Cole, “False Diagnosis and Needless Therapy of Presumed Malignant Disease in Women with False-Positive Human Chorionic Gonadotropin Concentrations,” Lancet, no. 355 (2000): 712–715.
16. JS Krouwer, “Using a Learning Curve Approach to Reduce Laboratory Error,” Accreditation and Quality Assurance, no. 7 (2002): 461–467.
