Medical Device Link.

IVD Technology Magazine
IVDT Article Index

Originally Published July 2000

Regulations and Standards

Traceability of assay calibrators: The EU's IVD Directive raises the bar

In the United States, IVD manufacturers have generally been sitting on the sidelines as new regulations and standards have emerged from Europe. But soon they will have to play in the game, and their readiness for the challenge will be tested.

Many firms believe that compliance with FDA's quality system regulation (Code of Federal Regulations, 21 CFR 820) will automatically convey compliance with the European Union's IVD Directive.¹ Such naïveté could lead to an unpleasant surprise. One of the most significant essential requirements of the IVD Directive appears to have escaped the attention of most U.S. manufacturers. This little-understood requirement of the directive pertains to traceability. It prescribes that calibrators and accuracy controls be traceable to available higher-order reference methods or materials.

In conjunction with this requirement, the International Organization for Standardization (ISO) has developed a new family of standards that dictate how manufacturers may demonstrate traceability. They are highly prescriptive, allowing little leeway. A manufacturer's ability to CE mark a quantitative assay depends on its ability to prove that its calibrators and controls have metrological traceability.

Traceability

Annex 1, section 3, of the IVD Directive lists this essential requirement:

The devices must be designed and manufactured in such a way that they are suitable for the purposes referred to in Article 1(2)(b), as specified by the manufacturer, taking account of the generally acknowledged state of the art. They must achieve the performances, in particular, where appropriate, in terms of analytical sensitivity, diagnostic sensitivity, analytical specificity, diagnostic specificity, accuracy, repeatability, reproducibility, including control of known relevant interference, and limits of detection, stated by the manufacturer. The traceability of values assigned to calibrators and/or control materials must be assured through available reference measurement procedures and/or available reference materials of a higher order.

The last sentence seems innocuous enough, until one reflects on the metrological concept of traceability.

Demonstrating traceability is not the same as demonstrating accuracy. Traceability means that a calibrator value is systematically derived from a higher-order reference material or reference method through an unbroken chain of comparisons. In other words, the value-assignment scheme must start with the highest available reference method or reference material—ideally a National Institute of Standards and Technology (NIST) standard traceable to the Système Internationale (SI), a World Health Organization (WHO) standard, or another certified reference material. Accuracy is transferred from these primary reference materials or methods, through secondary or intermediate reference materials and methods, to the product calibrators and, ultimately, to the measurement of patients' specimens. This, essentially, is metrological traceability as defined by ISO and as required by the IVD Directive.

Some manufacturers might think that an independent demonstration of accuracy would be sufficient to support a claim of traceability. In the case of serum cholesterol, for example, many firms have provided calibrators certified by the National Cholesterol Reference Laboratory Network administered by the Centers for Disease Control and Prevention (Atlanta). However, not even direct comparison of results from a routine method and a reference method will satisfy the traceability requirement. The regulation is concerned with the way accuracy is established and with the statistical uncertainty associated with the value assigned to the calibrator. U.S. manufacturers may have assumed that FDA's regulatory principle of substantial equivalence would cover the requirement. But substantial equivalence—even to an international reference method—is not the same as metrological traceability.

The new regulatory requirements for calibrator traceability have largely been driven forward by the comparatively greater importance northern European laboratory professionals place on accuracy, which they believe is essential to the transferability of patients' results from laboratory to laboratory and country to country. Wording in standard ISO/prEN 17511 spells it out: "It is essential that results reported to physicians and patients are adequately accurate (true and precise) to allow correct medical interpretation and comparability over time and space."² Whereas U.S. laboratories accept differences among methods as inevitable and compensate with method-specific reference ranges, their European counterparts have refused to abandon the pursuit of analytical truth. The IVD Directive has formalized this goal in regulation.

Whether traceable calibrators improve the clinical utility of an assay will continue to be debated in medical circles, but for the IVD industry the question is moot. The requirement is now firmly embedded in European regulation. Industry, which had its opportunity to object during the formative years of the IVD Directive, must now comply.

Traceability Standards and Metrology

To implement the IVD Directive, the European Commission of Health Ministers mandated development of a series of voluntary standards to promote consistent compliance with the directive's essential requirements. The charter to develop European IVD standards was given to the Brussels-based European Committee for Standardization (CEN), which has a cooperative arrangement with ISO in the standards-setting arena.

Conformance to an official European standard creates an automatic presumption of compliance with the pertinent essential requirements of the directive; therefore, following the European standards is the path of least resistance. The standards are voluntary only in the sense that manufacturers are free to choose a different, equally valid approach to meeting an essential requirement. But a manufacturer whose CE mark is challenged by a competent authority can expect an uphill battle if it has chosen an alternative route.

The national standards bodies that constitute the membership of CEN and ISO are heavily oriented toward metrology—the science of measurement—and clinical laboratory medicine involves measurement. The focus on traceability, an important metrological concept, attracted representatives from national metrology institutes and universities to the CEN and ISO committees formed to draft the new standards. In effect, the traceability requirement opened the door for metrologists to gain a foothold in laboratory medicine and influence the traceability standards.

Metrologists are principally interested in how accurately a substance can be measured. When the standards were being drafted, committee debates about metrological principles and terminology, while interesting on a philosophical level, often overshadowed concern for the intended clinical uses of the assays being regulated. This explains the academic tone of the resulting standards.

The series of traceability standards is anchored by ISO/prEN 17511, Metrological Traceability of Values Assigned to Calibrators and Control Materials. The basic concepts and model traceability schemes for IVD calibrators are detailed in this document. A companion standard for enzyme methods is ISO/prEN 18153.³

ISO has defined metrological traceability as an unbroken chain of comparisons, from higher- to lower-order materials and methods (See Figure 1). In the first step, an international primary reference method assigns a target value to an international primary reference material. Primary reference methods are the ultimate in accuracy: they are based on measurement principles proven to be analytically specific, have a low uncertainty of measurement, and provide direct traceability to an SI unit of measurement without reference to a calibrator. They are frequently available only in national metrology institutes, such as NIST, or in a few institutions accredited to run them.

The accuracy—or trueness in ISO terminology—of the primary reference method is transferred to the primary reference material. Since primary reference materials are expensive and primary reference methods impractical for routine use, intermediate calibrators and methods are needed to continue the transfer of accuracy to the product calibrators and, ultimately, to the patient specimens. Two intermediate transfer steps are shown in the figure.

Each transfer step increases the error, or uncertainty, of the final measurement. Therefore, the number of transfer steps in the traceability chain should be as few as practical. The uncertainty associated with each step is cumulative and incremental. This is shown graphically at the right side of Figure 1. In some cases it may be practical to assign values directly to the manufacturer's in-house working calibrator, eliminating the need for a secondary reference material and thereby reducing the final uncertainty. However, attempts to shorten the chain may be hampered by the lack of reference materials that are commutable with human serum. Consequently, additional intermediate steps that involve the use of panels of human samples may be required.

In a significant departure from current practice, manufacturers will have to account for the total uncertainty accompanying the calibrator values and will either have to provide the information in the product labeling or make it available to end-users on request. Knowing the uncertainty associated with a calibrator value is as important in metrology as knowing the calibrator value itself.

While physicians might be confused by such additional information, it is difficult to argue that laboratory directors cannot use it to make better-informed choices. For example: lot-to-lot accuracy shifts that affect many current immunoassays increase the uncertainty of calibrator values. Manufacturers that find ways to reduce it should thereby gain a competitive advantage. This kind of information is rarely available to customers today.

A better understanding of current international thinking on expressions of uncertainty can be obtained by reference to the Guide to the Expression of Uncertainty in Measurement (GUM).⁴ A U.S. document based on this guide is available on the NIST Web site (http://www.nist.gov).⁵

Filling Infrastructure Gaps

Primary reference methods and materials exist for only 25–30 clinical analytes, such as electrolytes, metabolites, steroid hormones, and thyroid hormones. These make up a large proportion of the routine tests in typical clinical laboratories.

Although most of the 400–600 clinically relevant analytes have not been standardized to this extent, WHO and other international organizations have developed reference materials and methods that can serve as the basis for a traceability chain. International conventional reference materials for analytes such as hemoglobin A1c have been assigned values by international conventional reference methods.

Approximately 30 analytes, such as hemostatic factors, have an international conventional reference method but no international conventional reference material. And about 250 analytes have no reference method but do have international conventional calibration materials with values assigned by a protocol based on international consensus. These include the WHO international reference preparations for protein hormones and other biological analytes.

Approximately 300 analytes remain without either reference method or reference material that could be used in a traceability chain. Tumor markers (e.g., CA-125) and antibodies (e.g., anti-HCV) fall into this category. Until such time as international standards are established, manufacturers are expected to establish in-house reference methods and calibrators to support value assignment for their product calibrators and controls. Once higher-order reference systems become available, manufacturers will be expected to use them as the basis for their calibration schemes.

The large number of analytes without any international basis for assigning calibrator and control values has caught the attention of national metrology institutes in Europe and the United States. Proposals to close the gaps are circulating on both sides of the Atlantic. Companies that manufacture assays without currently available reference materials or methods ought to consider getting involved in ongoing European initiatives, an upcoming needs-assessment workshop organized by NIST, or the possible development of a U.S. network of accredited reference laboratories.

Manufacturers now have the option to establish in-house reference laboratories, but, given the stringent quality system requirements of the applicable standards ISO DIS 15195 and ISO/IEC/FDIS 17025, many firms will probably find it advantageous to outsource their calibrator- and control-value assignment activities.^6,7 In addition to the ISO standards on reference measurement laboratories, two companion standards, one on reference measurement procedures and another on reference materials, have been written to complete the series of standards intended to support the IVD traceability requirement.^8,9 These also impose strict requirements for in-house reference materials and methods.

Outside reference laboratory services are offered extensively in Germany and are available in a few other European countries, but they are now available in the United States only from a few university laboratories, such as the Medical College of Wisconsin (Milwaukee), and a few commercial enterprises, such as the Pacific Biometrix Institute (Seattle).

Development of new reference methods and materials has been at a standstill in the United States for several years. The National Committee for Clinical Laboratory Standards, now NCCLS, established the National Reference System for the Clinical Laboratory (NRSCL) in the 1970s to "credential" national reference methods for U.S. laboratories. The system remained active while strong champions of reference systems were available, but interest waned in the 1980s as cost pressures on laboratory testing eliminated the resources necessary to develop and validate new reference methods. During the past decade, the NRSCL was subsumed into the NCCLS committee infrastructure where it has lain relatively dormant. With the need for credentialed reference methods and materials that has been created by the IVD Directive, interest in resurrecting the NRSCL is becoming apparent. The NCCLS consensus process could provide a sound basis for undertaking global leadership in this area.

The Language of Metrology

IVD manufacturers planning to continue marketing in Europe should already be studying the family of traceability and reference system standards discussed above. That most are not actively so engaged is partly attributable to the language barrier that separates metrology and the clinical sciences. The standards are very difficult to read and comprehend. However, that difficulty will not be an acceptable excuse for nonconformance.

To comply with ISO rules, which CEN has adopted, the standards had to be written in the international language of metrology. This language is laid out in the GUM and the Vocabulary of International Metrology (VIM).¹⁰ The VIM (specifically, VIM:1993) was prepared by a joint working group of experts appointed by the ISO, IFCC, International Bureau of Weights and Measures, International Electrotechnical Commission, International Union of Pure and Applied Chemistry, and International Union of Pure and Applied Physics.

Much of the terminology of metrology is unfamiliar to the U.S. IVD industry, which largely employs the NCCLS terminology used by its customers or the ICH guidelines for validating analytical procedures.^11-13 Familiar terms like accuracy, analyte, and interfering substance are missing from metrological language, or differently defined. These three examples become trueness, measurand, and influence quantity. Statistical error becomes uncertainty. At first glance such translations might seem arbitrary and unnecessary, but the reasons for standardizing terminology are persuasive. EDMA is currently writing a guidance paper on determining traceability that should help industry overcome the language barrier.

Conclusion

The European IVD Directive will not allow U.S. industry to stand still with respect to calibration and reference systems. The resource-pooling initiatives designed to bring about agreement on a single set of global standards are long overdue, and industry should wholeheartedly support these efforts. Although the transition period may be somewhat painful, as manufacturers, laboratories, and physicians adjust to numerous necessary calibration changes, the payoff that can be expected is greater stability as patients' laboratory data become anchored to global standards.

The current system, where each manufacturer attempts to calibrate its assays with methods and materials developed, validated, and maintained in-house, is highly inefficient. If a practical system of reference methods, reference materials, and reference laboratories were available to support manufacturers' value-assignment activities, the ongoing cost for any single manufacturer would be no greater than it is today, and might be lower owing to significant reductions in accuracy-related production waste, customer complaints, and recalls. These improvements would offer obvious benefits to customers as well.

References

1. "Council Directive 98/79/EC of the European Parliament and of the Council of 27 October 1998 on In Vitro Diagnostic Medical Devices," Official Journal of the European Union L331 (December 12, 1998).

2. In Vitro Diagnostic Medical Devices—Measurement of Quantities in Samples of Biological Origin—Metrological Traceability of Values Assigned to Calibrators and Control Materials, ISO/prEN 17511 (Geneva: International Organization for Standardization, 1999).

3. In Vitro Diagnostic Medical Devices—Measurement of Quantities in Samples of Biological Origin—Metrological Traceability of Values for Catalytic Concentration of Enzymes Assigned to Calibrators and Control Materials, ISO/prEN 18153 (Geneva: International Organization for Standardization, 1999).

4 . Guide to the Expression of Uncertainty in Measurement (Geneva: International Organization for Standardization, 1995).

5. Guidelines for Evaluating and Expressing the Uncertainty of NIST Measurement Results, NIST technical note 1297 (Gaithersburg, MD: National Institute of Standards and Technology, 1994).

6. Requirements for Reference Measurement Laboratories in Laboratory Medicine, ISO DIS 15195 (Geneva: International Organization for Standardization, 1999).

7. General Requirements for the Competence of Testing and Calibration Laboratories, ISO/IEC/ FDIS 17025 (Geneva: International Organization for Standardization, 1999).

8. In Vitro Diagnostic Systems—Measurement of Quantities in Samples of Biological Origin—Preparation of Reference Measurement Procedures, ISO DIS 15193 (Geneva: International Organization for Standardization, 1999).

9. In Vitro Diagnostic Medical Devices—Measurement of Quantities in Samples of Biological Origin—Description of Reference Materials, /ISO DIS 15194 (Geneva: International Organization for Standardization, 1999).

10. International Vocabulary of Basic and General Terms in Metrology, 2nd ed. (Geneva: International Organization for Standardization, 1993).

11. Terminology and Definitions for Use in NCCLS Documents; Approved Standard, NRSCL8-A3 (National Committee for Laboratory Standards, 1998).

12. Guideline for Industry: Text on Validation of Analytical Procedures, Document ICH-Q2A (International Conference on Harmonization, 1995).

13. Guidance for Industry: Text on validation of Analytical Procedures, Methodology, Document ICH Q2B (International Conference on Harmonization, 1996).

Donald M. Powers is an indepenent consultant (Mundelein, IL). The author wishes to thank Neil Greenberg, PhD, of Ortho-Clinical Diagnostics (Raritan, NJ) for his assistance in preparing this article.