REGULATIONS & STANDARDS
 |
Donald M. Powers, PhD, is president and principal consultant of Powers Consulting Group (Pittsford, NY) and is a member of IVD Technology’s editorial advisory board. He can be reached at powers@frontiernet.net.
|
Controlling Risks to Patients
A visit to a modern clinical laboratory provides ample evidence of the risk control practices that IVD manufacturers have employed: electronic monitoring systems, integrated procedural controls, fail-safe error messages, automated calibration, refrigerated onboard reagent storage, bar coded labels, and explicit warning stickers. Whether such risk control features originated from planned risk management activities or corrections of unexpected failures, many of these features were commonplace well before risk management came into vogue.4 Regardless of the motivation, their main purpose is to reduce the risks to patients from incorrect IVD test results.
As stated in the preamble to the U.S. quality system regulation, FDA expects manufacturers to reduce risks to an acceptable level.5,6 The European Union’s IVD Directive goes further by requiring manufacturers to reduce risks as far as possible, and to select risk control measures in the following order: inherent safety by design, protective measures in the device or the manufacturing process, and information for safety.7
The previous articles in this series discussed ways that IVD manufacturers can identify hazards and estimate their risks.2 If risk analysis determines that the risk from a possible failure mode is negligible, further risk reduction is not necessary. For the remaining risks, ISO 14971 outlines a logical sequence of risk control stages that manufacturers must follow to conform to the standard (see Figure 1). These stages are intended to be checkpoints to promote systematic decision-making and ensure that IVD manufacturers document their risk management decisions for future reference. A justification for each decision must be included if the rationale is not obvious.
 |
Figure 1. Risk control stages. Source: ISO 14971:2000.
|
Analyzing Risk Control Options
The first stage is a comprehensive analysis of risk control options, which enables IVD manufacturers to select the most appropriate control measures for each specific risk. Manufacturers can do little to reduce the severity of harm, since that depends mainly on the medical actions taken in response to the IVD test results. For most IVDs, manufacturers can only reduce the probability that harm will occur.
ISO 14971 adopted the three-level hierarchy of risk control options from the IVD Directive, thereby placing the emphasis on mitigating risks through design whenever practicable. To show that the risk control measures were selected according to the prescribed priority hierarchy, IVD manufacturers must document that the options available to reduce each risk were identified, classified according to the hierarchy, and evaluated for technical and economic feasibility.
Several standards from the International Organization for Standardization (ISO; Geneva), the International Electrotechnical Commission (IEC; Geneva), the European Committee for Standardization (CEN), and the Clinical and Laboratory Standards Institute (CLSI; Wayne, PA) address safety aspects of IVDs. A few standards (e.g., IEC’s new usability standard for medical devices) integrate elements of the risk management process.8 When appropriate,
IVD manufacturers should consider such standards as options to control identified risks, since conforming to a recognized standard generally conveys a presumption of risk acceptability.
Inherent safety by design for an IVD assay means its performance characteristics and reliability meet the medical requirements of its intended use. The components that affect specificity, precision, stability, and traceability are the keys to improving analytical performance, while sound hardware and software design features can prevent instrument-generated spurious results and enhance the uptime of electromechanical systems. In addition, automation can eliminate the potential for human error, bar coding can ensure the proper identification of patients, samples, or reagent lots, and attention to human factors can simplify procedures and eliminate causes of error.
Protective measures reduce the probability that incorrect results will be reported to physicians or patients. Examples of protective measures include fail-safe software, which results in error messages instead of incorrect results, and fault-tolerant systems, which continue to operate correctly after a fault occurs. Instrument-based controls are now commonplace in IVD systems, and quality control testing has been integrated into many assays.9 Other examples of protective measures are in-process tests that detect defects introduced in the manufacturing process and final-release tests that verify conformance to performance specifications.
Information for safety is the labeling, which includes operating instructions and other information to avoid risks. Historically, IVD manufacturers have relied heavily on labeling to help clinical laboratories avoid generating hazardous results. However, adding new instructions or precautions without first attempting to eliminate the failure causes or to incorporate protective mechanisms is contrary to the philosophy of ISO 14971. The standard reminds manufacturers that “information for safety is the least preferred method of risk control measure, to be used only when other risk control measures have been exhausted.” Risk management experts allow a risk reduction of no more than one level on the probability scale from labeling or training. Although mandatory training programs designed to reduce risks can sometimes achieve greater risk reduction, training alone is seldom effective in eliminating errors.10
IVD manufacturers often overestimate the ability of laboratory quality control (QC) procedures to detect incorrect test results. When delegating risk controls to clinical labs, design engineers must make assumptions with caution. While a manufacturer’s QC recommendations may provide helpful advice, there is no assurance that labs will follow them. In order to qualify as a protective measure, a QC procedure would have to be an integral part of the assay procedure. The IVD guidelines in annex H of the second edition of ISO 14971, which has recently been advanced to a final draft international standard, make it clear that an IVD manufacturer’s recommended QC procedures, as well as recommended confirmatory tests, are only considered information for safety.11
Warnings also have limited effectiveness as a risk reduction measure and should be considered as a last resort. The pressure to avoid calling attention to an assay’s shortcomings in a package insert has all too often led to vague, innocuous statements that would not pass an objective test for effectiveness. The revised ISO 14971 will include a new annex titled “Information for Safety and Information about Residual Risk,” which provides guidance on using information for safety as a risk control measure and promoting true awareness when disclosing residual risks.11
Implementing Risk Control Measures
The risk control implementation stage begins once an IVD manufacturer decides which options yield the greatest risk reduction. Often, more than one risk control approach is needed. Manufacturers have to verify each risk control measure twice, first to document that it was implemented correctly, and then to verify or validate its effectiveness in reducing the risks to the desired level.
IVD manufacturers should set up risk management documentation to ensure clear traceability from each identified hazard and hazardous situation to the associated risk control measures. While manufacturers are able to verify some hardware- and software-based risk controls individually through fault injection techniques, a means of testing and evaluating fault-tolerant systems,12 most will demonstrate the cumulative risk reduction from multiple control measures during design validation.
Evaluating and Disclosing Residual Risks
Risks are rarely eliminated completely. Risks that remain after a manufacturer has implemented all risk control measures are called residual risks, which must meet the acceptability criteria in the risk management plan. Confirming that the residual risks from each hazardous situation have been reduced to an acceptable level is a pivotal checkpoint in the risk control process.
If the acceptability criteria are met, then manufacturers must determine what information to include in the labeling to disclose the residual risks. They must also consider the comprehension level of end-users and the detail that users need in the labeling to make informed decisions. If the acceptability criteria are not met, then risk reduction efforts continue until further reduction is not technically or economically feasible. If the risk acceptability criteria are still not met, an IVD manufacturer must either abandon the project or justify it based on a risk-benefit analysis.
ISO 14971 recognizes that end-users need to be aware of the residual risks in order to manage them. Such disclosure involves providing the information necessary to understand the residual risks and minimize exposure to the hazards.
In IVD labeling, the information in the “Limitations of the Method” section is an example of residual risk disclosure. This part of the labeling contains information such as drug interferences that the IVD manufacturer could not eliminate. Disclosure of drug interferences is not a risk control, since laboratories do not have any practical means to monitor the presence of drugs in specimens. However, in the spirit of ISO 14971, IVD manufacturers should opt to disclose these residual risks only after their efforts to eliminate the interferences prove unsuccessful. The information in the “Performance Characteristics” section is another example of residual-risk disclosure. With such information, a lab director can make informed decisions about whether a particular assay will be useful for its intended medical purpose.
The details about how IVD manufacturers control specific risks are becoming more important with the advent of equivalent quality control.13,14 This concept allows a laboratory to build on a manufacturer’s risk controls and design a holistic quality control program. This requires knowing not only the circumstances that can lead to incorrect results but also the internal control mechanisms that can detect or suppress them.15,16
Although in principle an IVD manufacturer is responsible for deciding what information is necessary to include in the product labeling, in practice much of the content has already been decided by labeling regulations and standards.17–20 Several examples of prescribed information for safety are intended to help laboratories avoid common use errors (see Table I).
 |
Table I. Information for safety to avoid potentially hazardous use errors. Source: ISO/DIS 14971:2005, annex H.
|
Performing Risk-Benefit Analysis
A risk-benefit analysis is required when efforts to reduce residual risks to acceptable levels have failed, yet the IVD manufacturer believes the product is still worthwhile. Unfortunately, while methods for estimating risks with reasonable confidence exist, a standardized approach to estimate the benefits has not been developed. ISO 14971 acknowledges that risk-versus-benefit decisions are largely matters of judgment by experienced and knowledgeable individuals. Some guidance for conducting risk-benefit analyses is included in annex D.6 of the draft second edition of ISO 14971.
Risk-benefit analyses of individual residual risks are difficult and provide at best only enough information to decide whether to proceed with a design project. Having reduced each individual risk in an iterative manner until it is either acceptable or irreducible, any residual risks that remain outside the acceptability criteria require a decision whether to proceed with the project. If it is obvious that an individual risk is irreducible and will jeopardize an IVD manufacturer’s ability to meet the overall acceptability criteria or to provide a product with medical benefits that outweigh its overall residual risks, the development project may be stopped. Outside clinical consultation often helps in making this judgment. If the answers are not obvious, the project will usually continue.
Addressing Risks Resulting from Risk Control Measures
Occasionally, implementing risk control measures creates new hazardous situations. For example, a proliferation of warning labels can confuse laboratory operators and desensitize them to hazard warnings. Similarly, an overly complicated set of instructions may be difficult to follow without errors. There are also abundant examples in the recall literature in which actions intended to reduce risks inadvertently created new hazards. IVD manufacturers must review the results of the risk control phase to avoid increasing risks.
In some cases, IVD manufacturers may introduce a new hazardous situation intentionally as the lesser of two risks. For example, an instrument programmed to give an error message when a fault occurs is trading off the risk from a delayed result against the risk from a possible incorrect result. The actual degree of risk depends on how the test results are used. For self-testing glucose monitors, if a diabetic were not able to perform blood glucose tests in a timely manner because the meter malfunctioned, there is a chance of accidental overmedication with insulin, leading to hypoglycemic shock. While an error message may seem clearly preferable to an incorrect glucose result, manufacturers must estimate the risks created by a delay in availability of the results and evaluate them against the risk acceptability criteria.
On the other hand, a delay in the availability of routine hospital laboratory test results while an automated instrument undergoes repairs is less likely to cause serious harm. Labs anticipate the need for service calls and are expected to have backup plans for lab tests that are urgent. Acceptability of the risks from delayed results is determined in each case by a risk assessment.
Verifying Completeness of Risk Control
A verification checkpoint in the risk control process was added as a safeguard against possible overlooking of any necessary risk control measures in the rush to launch a new product. IVD manufacturers are required to verify that they have adequately addressed all hazardous situations identified in the risk analysis phase and any new ones created by the risk control measures.
ISO 14971 also requires manufacturers to record the results of all risk control activities in the risk management file, including each decision to take or not take action, every review and verification check discussed above, and the results of the risk reduction activities. This requirement reinforces the need for IVD manufacturers to establish a logical documentation system at the outset, with the necessary traceability to show that they have addressed all hazardous situations and reduced all risks to acceptable levels.
Evaluating Residual-Risk Acceptability
Even after all the individual risks have been mitigated, there is still the question of whether the overall risk of using the IVD device will be acceptable. After documenting the reduction of each individual residual risk, an IVD manufacturer compiles the information in a way that reveals the overall residual risk. The overall residual risk is the risk that remains after a manufacturer has implemented all risk control measures. The risk of using the device may not be acceptable because of the cumulative probability of serious harm from many low-probability failure modes. In that case, an IVD manufacturer must apply further risk control measures to make the overall residual risk acceptable.
Estimating overall residual risk is more challenging than estimating risks from specific device failures, and up to now, little guidance has been available to help IVD manufacturers in evaluating it. A discussion of several possible approaches has been added to the second edition of ISO 14971 as annex D. Two tools described in this annex are particularly applicable to IVDs. In event tree analysis, multiple individual risks from a specific sequence of events can be considered together to determine if the overall residual risk is acceptable. In fault tree analysis, the combined probability of harm from several hazardous situations can be estimated from their individual probabilities. The output of a cause-consequence analysis, a blend of fault tree and event tree analysis, is a diagram for documenting and communicating relationships between risks and their initiating causes.21
For well-understood IVD assays, confirming that the overall residual risk is acceptable by performing a detailed risk-by-risk comparison to a comparable assay on the market may suffice. If manufacturers use this approach, they must ensure that information about risks of the comparable assay is up-to-date. Current product labeling and performance data can often be obtained from manufacturers’ Web sites or from clinical laboratories. Peer-reviewed data from product evaluations and clinical experience are found in laboratory journals. In addition, a wealth of adverse-event experience, recalls, and summaries of product safety and effectiveness is available on the FDA Web site.22
Another approach especially useful for evaluating the overall residual risk of new IVD systems is to engage application specialists (i.e., laboratory technologists) who are familiar with using similar devices. An important caveat is that they must not have been directly involved in developing the device being evaluated.
The application specialists evaluate the system in a representative clinical lab environment to confirm the acceptability of its overall residual risk, specifically considering the characteristics related to safety identified in the risk analysis (e.g., performance, usability, and reliability). In addition, the specialists review the operating instructions and other essential labeling information to identify instructions that are difficult to follow, excessive reliance on warnings, and conflicting requirements. Based on their professional knowledge and experience, they would judge whether the overall residual risk of the system is acceptable.
What if the overall residual risk is judged not acceptable according to the criteria in the risk management plan, and further risk reduction is not practicable? An IVD manufacturer has to decide whether the potential medical benefits outweigh the overall residual risk.
Risk-benefit analyses of overall residual risk are not as common for IVDs as they are for life-sustaining medical devices. Manufacturers must spend considerable energy in developing a defensible comparison of overall residual risk with the overall benefit that the IVD device provides. Quantifying the medical benefits of an IVD assay with an unacceptable potential for incorrect results is difficult. When the overall residual risk falls outside management’s risk acceptability criteria, IVD manufacturers usually abandon plans to commercialize the device.
If an IVD manufacturer pursues a risk-benefit analysis, the decision process is similar to that used for analyzing the individual risks. If the evidence supports a conclusion that the medical benefits outweigh the overall residual risk, the overall residual risk is judged acceptable. Otherwise, the manufacturer has no other choice but to abandon the product.
Conclusion
A key element of the last IVD design review prior to a new product launch is a comprehensive review of all the risk management activities that have been undertaken. An IVD manufacturer must verify three things: proper execution of the risk management plan, acceptability of the overall residual risk, and the entire organization’s readiness to carry out its ongoing risk management duties during the life of the device. The documented summary of this review becomes the high-level risk management report required for conformance to ISO 14971.
The revised ISO standard no longer requires the risk management report to document traceability of hazards to risk controls. This requirement was eliminated because it caused the final report to become unwieldy, particularly for complex systems that require extensive risk analyses. Nevertheless, IVD manufacturers must still demonstrate traceability in the risk management file. A guidance document from the Global Harmonization Task Force presents an example of an efficient risk management summary table that IVD manufacturers can use to document how the individual risks were controlled for a given device, and the status of implementation and verification of the risk control measures.23 Such a document will be an invaluable tool for continuing to manage risks through a product’s life cycle.
The next installment of this series will discuss the production and postproduction monitoring stages of the risk management process, and explore how these aspects of the risk management process can be integrated into a quality management system. That article will also highlight common gaps in the risk management programs at IVD companies.
1. DM Powers, “Risk Management for IVDs, Part 1: Planning and Documenting the Risk Management Process,” IVD Technology 12, no. 2 (2006): 28–33.
2. DM Powers, “Risk Management for IVDs, Part 2: Assessing Risks to Patients from Incorrect Test Results,” IVD Technology 12, no. 3 (2006): 24–31.
3. ISO 14971:2000, “Medical Devices: Application of Risk Management to Medical Devices” (Geneva: International Organization for Standardization).
4. DR Wallace and DR Kuhn, “Lessons from 342 Medical Device Failures,” National Institute of Standards and Technology Web site (Gaithersburg, MD: 1999 [cited 23 March 2006]); available from Internet: http://hissa.ncsl.nist.gov/wallace/hase99.pdf.
5. Federal Register 61 FR:52601–52662, October 7, 1996.
6. FDA, “Guide to Inspections of Quality Systems,” FDA Web site (Rockville, MD: August 1999 [cited 23 March 2006]); available from Internet: www.fda.gov/ora/inspect_ref/igs/qsit/QSITGUIDE.PDF.
7. “Directive 98/79/EC of the European Parliament and of the Council of 27 October 1998 on In Vitro Diagnostic Medical Devices,” Official Journal of the European Communities L331 (1998).
8. IEC/CD 62366, “Medical Devices: Application of Usability Engineering to Medical Devices” (Geneva: International Electrotechnical Commission, 2004).
9. FD Lasky, “Technology Variations: Strategies for Assuring Quality Results,” Laboratory Medicine 36, no. 10 (2005): 617–620.
10. ME Maddox, “Human Factors: Designing Medical Devices to Minimize Human Error,” Medical Device & Diagnostic Industry 19, no. 5 (1997): 166–178.
11. ISO/DIS 14971:2005, “Medical Devices: Application of Risk Management to Medical Devices,” 3rd ed. (Geneva: International Organization for Standardization).
12. MC Hsueh, TK Tsai, and RK Iyer, “Fault Injection Techniques and Tools,” Computer 30, no. 4 (1997): 75–82.
13. MB Bandy, “CLIA Quality Control Requirements,” IVD Technology 11, no. 2 (2005): 24–29.
14. DM Powers and T Roscoe, “QC for the Future,” IVD Technology 11, no. 9 (2005): 22–27.
15. L Ochs, “QC for the Future: CLSI Standard Development and Option 4 Proposal,” Laboratory Medicine 36, no. 10 (2005): 639–640.
16. DM Powers, “Laboratory Quality Control Requirements Should Be Based on Risk Management Principles,” Laboratory Medicine 36, no. 10 (2005): 633–638.
17. Code of Federal Regulations, 21 CFR 809.10.
18. EN 375:2001, “Information Supplied by the Manufacturer with In Vitro Diagnostic Reagents for Professional Use” (Brussels: European Committee for Standardization).
19. EN 591:2001, “Instructions for Use for In Vitro Diagnostic Instruments for Professional Use” (Brussels: European Committee for Standardization).
20. ISO 18113, “Clinical Laboratory Testing and In Vitro Diagnostic Test Systems—In Vitro Diagnostic Medical Devices—Information Supplied by the Manufacturer (Labeling)” (Geneva: International Organization for Standardization).
21. Center for Chemical Process Safety, Guidelines for Hazard Evaluation Procedures with Worked Examples, 2nd ed. (Hoboken, NJ: Wiley, 1992).
22. “Databases on the FDA Website,” FDA Web site (Rockville, MD: [cited 23 March 2006]); available from Internet: www.fda.gov/search/databases.html.
23. “Implementation of Risk Management Principles and Activities within a Quality Management System,” Global Harmonization Task Force Web site (Brussels: 2005 [cited 23 March 2006]); available from Internet: www.ghtf.org/sg3/inventorysg3/sg3n15r82005.pdf.
