Medical Device Link.

REGULATIONS & STANDARDS

Donald M. Powers, PhD, is president and principal consultant of Powers Consulting Group (Pittsford, NY) and is a member of IVD Technology’s editorial advisory board. He can be reached at powers@frontiernet.net.

The first article of this series (IVD Technology, March 2006) introduced the principles of risk management, discussed the significance of risks from IVD devices, and emphasized the need to incorporate risk management planning into the product realization process. This second article will focus on the risk assessment phase of risk management.

According to ISO 14971, risk assessment consists of risk analysis and risk estimation.¹ Risk analysis is the systematic use of available information to identify hazards and the sequences of events that lead to hazardous situations. Risk estimation determines the level of risk from probability that harm will occur and the severity of that harm.

Hazard Identification

The hazards inherent in IVD products, including failure modes that cause IVDs to become hazardous, must be understood because they are the starting points for the chains of events that can lead to harm. As early as possible during the design phase, IVD manufacturers should develop lists of potential hazards to patients, users, and the environment. Doing so at this time provides the most cost-effective opportunity to minimize the occurrence of hazards or avoid them altogether through design improvements.

Hazard identification starts by examining the intended uses for the IVDs, and any foreseeable use errors. The most common hazard to patients from IVD products comes from clinically incorrect test results. Erroneous results create hazardous situations for patients if they are plausible values and are reported to physicians. For some analytes, delayed availability of test results can also be hazardous. By starting with an understanding of the immediate patient hazards (i.e., incorrect or delayed results), IVD manufacturers can work backward to identify possible failures that could cause or contribute to their occurrence, and then find ways to prevent, detect, or control them.

IVD hazards are defined by the medical uses of the test results. The severity of possible harm to patients depends on the particular analyte. Every test has key performance requirements derived from its intended clinical use, such as accuracy and precision; specificity for quantitative assays, and diagnostic sensitivity and specificity for qualitative assays. Qualified medical input is essential so that the medical uses of the test results are understood, including the performance characteristics required for medical decisions and the potential harm that could arise from a misdiagnosis.

Figure 1. (click to enlarge) A risk model for laboratory-use IVD devices. Source: ISO/FDIS 14971:2006.

ISO/FDIS 14971:2005 provides a risk model for laboratory-use IVD assays (see Figure 1).² For example, if a glucose analyzer malfunction led to a falsely high result, a physician might conclude that a patient is hyperglycemic and decide to administer insulin. Such a diagnosis would create a hazardous situation if the patient were actually hypoglycemic.

Risk Analysis Team

Once IVD manufacturers identify the hazards associated with their products, they can analyze the effects of the hazards and estimate the risks through risk analysis. A team of subject matter experts with collective knowledge of the product, its manufacturing process, the laboratory (or other) use, and its clinical application conducts the risk analysis. Team members might consist of research scientists, manufacturing engineers, quality engineers, customer support specialists, clinical affairs specialists, and a facilitator skilled in the risk analysis techniques being applied. The team is charged with identifying the ways an IVD product can fail, determining the likely adverse effects of the failures, and recommending ways to prevent or detect the failures.

Reliability engineering techniques from the aerospace, defense, and other industries have been successfully applied to risk analysis of medical devices. The ISO standard does not prescribe any particular risk analysis methodology. In fact, the ISO 14971 authors left the choice of appropriate risk analysis techniques to the manufacturers.

Risk Analysis Tools and Techniques

Table I. (click to enlarge) Applicability of risk analysis techniques to different stages of risk analysis.

Risk analyses are iterative during the design and development stages, and continue to be refined throughout the product life cycle. Many different risk analysis methods and techniques can be used for IVD products (see Table I).³ Depending on the particular stage of development, the nature and complexity of the product, and the degree of experience that the IVD company has with similar products, it may be necessary to use different risk analysis tools, or combinations of tools. Although most IVD companies gravitate toward using failure modes and effects analysis (FMEA), or a variant called failure modes, effects, and criticality analysis (FMECA), other risk analysis tools may also be appropriate.

Preliminary risk analysis (PRA) and preliminary hazard analysis (PHA) are qualitative techniques that involve a disciplined analysis of potentially hazardous events. After identifying the potentially hazardous events, each hazard is analyzed separately to identify possible preventive measures. The results of the analysis provide a basis for determining which categories of hazards should be examined more closely.

FMEA and FMECA are bottom-up techniques that examine how component failures can affect a larger system. They are designed to identify potential failure modes for a product or process, estimate the consequences associated with each failure mode, and identify actions to reduce unacceptable risks to acceptable levels. This approach includes a method for evaluating the risks associated with the potential failure modes, such as risk priority numbers or criticality analysis. While both failure modes analysis tools can be very useful, if they are applied to improper situations they can become overly detailed and time-consuming. A recent article describes several inherent traps and provides helpful insights on the use and misuse of FMEA.⁴

Fault tree analysis (FTA) is a top-down method that starts by assuming a main system failure and analyzes what could cause it. FTA is more efficient than either failure modes analysis technique for analyzing combinations of failure events and human-use errors. FTA and failure modes analysis are often used together to evaluate complex systems for a comprehensive top-down and bottom-up risk analysis.

Event tree analysis (ETA) provides an inductive approach to analyzing the consequences of a failure or other undesired events. Construction of an event tree begins with an initiating event. The consequences of this initial event are followed through a series of possible paths that are affected by safeguards and external influences. Each path is assigned a probability of occurrence that enables the probability of the possible outcomes to be calculated. Specific combinations of events that lead to harm can then be analyzed by FTA.

Hazard and operability studies (HAZOP) consider possible deviations from the normal state of a manufacturing processes and identify the appropriate safeguards to prevent such deviations. HAZOP uses special guidewords—adjectives (e.g., more, less, no) combined with process conditions (e.g., speed, flow, pressure)—to identify scenarios that may result in a hazard, the consequences of the hazard, and measures to reduce its frequency.

Hazard analysis and critical control points (HACCP) can identify the steps in manufacturing processes that require controls because of their tendency to fail or the difficulty in detecting failures when they occur. HACCP builds on other risk analysis tools, which allows the risks to be estimated and controlled at each process step that entails significant risk.

Choosing a Risk Analysis Method

Information on these risk analysis procedures is available in textbooks, standards, and other sources.^5–10 Choosing the right method for the right situation is the key to a successful risk analysis. The factors that IVD manufacturers must consider when selecting an appropriate risk analysis tool include the following: the reasons for the risk assessment, the type of results needed, the resources available, the current stage of the product life cycle, the complexity of the system or process, the quality and timeliness of information available to the team, and the type of incidents targeted.

Some risk analysis techniques are more appropriate during the later stages of product development when more information is available. For example, if a risk analysis is needed for the proposed design of a new IVD, detailed descriptions, written procedures, and design drawings will not likely be available. This stage of the product life cycle would call for a less-detailed analysis technique than a failure modes analysis.

The quality and timeliness of existing documentation are also factors to consider. A risk analysis using out-of-date information would be a waste of time and resources. If an IVD manufacturer uses a risk analysis technique that relies on design documentation, the drawings must be up-to-date and exist in a suitable form.

Certain risk analysis techniques are better than others in specific situations. For example, FMEA is usually the best approach for analyzing an electromechanical system, while HAZOP does not work well for evaluating those types of systems. While FMEA is useful for analyzing single-fault events, but not human error or multiple events, FTA is better for analyzing human errors and combinations of failure modes, and is often used first to focus FMEA on specific components of a system.

Some risk analysis techniques get bogged down when analyzing complicated systems. For example, the effort required to perform FMEA is proportional to the number and types of events and effects being evaluated. Therefore, FMEA will take five times longer for a system containing 100 components than for a system containing 20 components. IVD manufacturers must consider the complexity and size of a system or process, the number of pieces of equipment and process steps, and the number and types of events and effects being analyzed.

In addition, IVD manufacturers should use more-thorough risk analysis techniques for those systems expected to involve significant risk and for those situations in which failures are expected to have severe consequences. A more-thorough risk analysis approach increases the chances that manufacturers will uncover possible problems.

As the development of an analyzer or assay progresses and its design improves, more-detailed FMEA and FTA focused on the design are added to the initial hazard analyses, and the risk estimates are further refined. Eventually, the manufacturing process is analyzed to identify failures that can result in hazardous products. At each stage, the risk management file is updated with new information.

Risk Analysis Process

The goal of a risk analysis is to gather the proper amount of information that management needs to make decisions regarding potential hazards. This goal dictates the depth of risk analysis needed to obtain such information. Unnecessary risk analysis activities do not benefit decision makers, but instead use up time and money that could be better spent designing the product or solving problems.

The key is to strike the right balance by starting risk analyses at a general level and performing more-detailed analyses when additional information is needed for decisions about the project. Risk analysis should start with hazard identification, progress through a risk screening phase, and then focus on a detailed analysis of the important risks. This strategy filters out the unimportant issues as early as possible and ensures that the most important issues receive sufficient attention. The risk analysis should end as soon as enough information for decision making is available. Certain tools are useful at different stages of risk analysis (see Table I).

After hazard identification, the next step in risk analysis is risk screening. As the name suggests, risk screening provides general conclusions that broadly describe the risks. It is useful for identifying important risk areas and eliminating the remainder from further consideration. This step in the risk analysis process may provide enough information about intolerable risks that cannot be reduced with existing technology at an acceptable cost. However, moredetailed risk analyses are usually needed to evaluate IVD assays.

When risk screening indicates that an IVD product has significant or uncertain risks, structured tools can identify specific equipment faults, process failures, human errors, and combinations of events that might lead to adverse consequences. Such detailed, focused analyses characterize the risks qualitatively or quantitatively so that appropriate risk management strategies can be defined. To ensure the validity of the outcome, personnel that perform risk analyses must have training and experience in the particular technique being used.

Once the hazards are understood and the probabilities of occurrence estimated, a detailed risk assessment is needed to determine if the risks are acceptable by evaluating them against the IVD company’s risk-acceptance policy. The company management is responsible for establishing the criteria for acceptable risk, based on current values in society, and deciding when medical benefits outweigh the risks.

Clinical Risk Factors

Not every malfunction of an IVD product causes or contributes to harm to patients. Many quality control safeguards have evolved in laboratory medicine to discover incorrect test results before they are reported. Physicians have learned to cope with occasional erroneous results, most of which emanate from pre- and postanalytic errors. It is tempting to factor in the probabilities that erroneous results will be detected and not reported by labs, and dismissed by physicians as inconsistent with clinical impressions. In the parlance of risk management, a hazardous test result only develops into a hazardous situation when a patient is exposed to possible harm.

That being said, IVD manufacturers should err on the side of caution when estimating the probability that labs or physicians will detect believable but incorrect test results. The probability an erroneous result will be detected by activities outside the manufacturer’s control is difficult to estimate and justify objectively. While some laboratories may have highly sophisticated quality control systems, other less-fortunate labs have only rudimentary capabilities. Except for easily detectable systematic errors that develop slowly over time (e.g., reagent or control-fluid instability) or results clearly recognizable as incompatible with the patient’s condition, IVD manufacturers should assume that incorrect test results will be reported by the laboratories and will influence a physician’s decision. Risk assessments should be based on the worst-case scenarios.

With some IVD assays, a percentage of medically incorrect test results may occur during normal use, results that could lead to a wrong diagnosis or treatment. Such is the nature of analytical laboratory methods. James Westgard, PhD, of the University of Wisconsin Medical School (Madison, WI) evaluated the accuracy of several IVD assays against proficiency testing criteria and concluded that few assays operate at the six-sigma level, a quality standard that represents approximately 3.4 errors per million test results.¹¹ An error was defined as a test result that exceeds the allowable error limit established by the Clinical Laboratory Improvement Amendments of 1988 (CLIA) for proficiency testing results. Westgard calculated that cholesterol, calcium, glucose, and prothrombin time analyses perform between the three- and four-sigma level (an inherent error rate of 6210 and 233 per million test results, respectively). Glycohemoglobin and prostate specific antigen (PSA) tests perform at less than a three-sigma level.¹²

IVD manufacturers must analyze the risks associated with incorrect test results in normal use, and evaluate such risks against the risk acceptability criteria during assay design and development. Normally, the development of an IVD assay will stop if its performance cannot meet the medical utility requirements that are established as design inputs. If the medical demand is sufficient, the manufacturer may perform a risk/benefit analysis to determine if the benefits outweigh the risks.

Conclusion

When initiating a risk analysis, IVD manufacturers should evaluate each failure of their products to meet specified or implied performance characteristics as a possible hazard. Every IVD device will inevitably fail to meet one or more of its performance claims at some point. Such failures could be caused by a multitude of factors, including design flaws, manufacturing defects, improper transportation, inappropriate handling or storage, or use errors by laboratorians. Incorrect test results may occur as a consequence.

While it may seem unfair to require IVD manufacturers to consider the consequences of how their products are used when outside of their control, the overriding objective of risk management is to protect patients from possible harm. The manufacturers are expected to identify any foreseeable uses of their products that could be hazardous, and at a minimum inform users of the risks. The laboratories bear the responsibility for using the products according to the IVD manufacturers’ instructions and claims, especially when the manufacturer has made the risks obvious. In addition to the laboratory uses of the products, IVD manufacturers may have to consider reasonably foreseeable off-label uses of test results by physicians in the risk analysis.

The next articles in this series will discuss the risk control and postproduction monitoring stages of the risk management process, and explore how these stages can be integrated into the quality management system. These articles will also highlight common gaps found in the risk management programs at IVD companies.