Medical Device Link.

REGULATIONS AND STANDARDS

Jan S. Krouwer, PhD, is president of Krouwer Consulting (Sherborn, MA), which provides statistical and reliability consulting to the IVD industry. He can be reached at jan.krouwer@ comcast.net.

As a regulatory requirement for IVD devices, risk management is performed for all IVD products. However, shortchanging a risk management task for the sake of speed or due to a lack of manpower can have dire financial consequences and can lead to possible regulatory sanctions. IVD manufacturers only need to read recent FDA recalls to grasp fully the importance of risk analysis and evaluation.¹ In one case, the effect of a medical device product recall made financial news headlines.²

ISO 14971 provides guidance on risk management for medical devices, with an appendix written for IVD devices.³ This article provides 10 tips for effective risk management. (Avoiding the use of the risk priority number (RPN) is not included, since this tip was previously covered in depth.⁴) The tips discussed in this article complement a series of articles on risk management published in IVD Technology.^5–9

Tip 1. Challenge the Design, Don’t Just Document It

Table I. (click to enlarge) An FMEA fragment.

The failure modes and effects analysis (FMEA) fragment in Table I, which is conducted as part of a risk analysis, describes a control measure for preventing IVD instrument failure. The problem is that adding the thermistor has little to do with this FMEA, but rather was part of the original product design work. It may have even been part of the design specifications. Adding a thermistor as a control measure is an expected good engineering practice. IVD manufacturers must do a lot of work to document what exists and this FMEA documentation can be time-consuming. However, manufacturers should resist the temptation to end the FMEA after the documentation step and before the real value of an FMEA is realized.

Table II. (click to enlarge) An FMEA fragment (continued).

An example of how this FMEA fragment could be continued is shown in Table II. While the first row documents what has been done, the second row challenges the design. Any control measure can be questioned, but not every challenge results in a new control measure. Another possible challenge is to ask whether there are other ways this step can fail.

Another suggestion for FMEA is to review the control measure column and note how often “no control measure needed” appears. If the answer is never, there may be a potential problem, because there is always an acceptable level of risk. For example, IVD manufacturers can consider whether a double-check step can fail. While an additional check is perhaps warranted, at some point, no further control measure is needed.

In addition, FDA and other regulatory authorities are not the only source of regulatory requirements. Many IVD companies have internal regulatory requirements and need to keep this in mind when conducting risk analysis.

Tip 2. Distinguish between Potential and Observed Errors

Table III. (click to enlarge) The advantages and difficulties of the FMEA, FTA, and FRACAS technical methods.

Fault tree analysis (FTA) and FMEA reduce the risk of potential errors. A failure reporting and corrective action system (FRACAS) reduces the recurrence of observed errors. (Tip 10 discusses the difference between FRACAS and CAPA). Table III shows that each technique has its advantages and difficulties, and illustrates why both should be conducted before the product’s release.

While FMEA and FTA can start as soon as the design work begins, FRACAS starts after the design process begins, when many observed failures populate a FRACAS database before the product’s release. Quality assurance managers sometimes discount FRACAS because there is an emphasis to do it right the first time and avoid testing in quality. However, the knowledge does not exist to prevent design failures for complex IVD assay systems, and FRACAS can be the most efficient way to reduce risk.¹⁰ Moreover, if observed failures have been corrected before the product release, potential failures after the release have also been addressed.

Tip 3. Construct Quantitative Failure Rate Goals

A meaningful failure rate goal has to be quantitative, measurable, and significant to an IVD company’s business plan. For example, the rate of unscheduled service calls was for a large analyzer system used with modeling to combine failure modes into one overall goal.¹⁰ Some manufacturers might find the rate of recalls appropriate for smaller systems. Even though establishing a meaningful failure rate goal may seem obvious, some inadequate goals include the following:

• No goal.
• A “zero” goal (i.e., zero software failures or zero recalls). Undesirable events have probabilities of occurrence (FMEA, FTA) or actual rates of occurrence (FRACAS). Even an observed rate of zero has a confidence limit greater than zero.
• Unattainable goals. While “stretch” goals may motivate people, they can also be ignored.
• Too many goals. If an IVD manufacturer has 28 failure modes, generating 28 meaningful goals will be too complex.

IVD manufacturers that conduct FRACAS are less likely to have a problem establishing meaningful goals. Without FRACAS, failure rates could be estimated using quantitative FTA. However, this practice is unlikely to occur for IVD products due to its complexity.

Tip 4. Embrace Failure-Detection Recovery Schemes

Figure 1. (click to enlarge) The failure—detection— recovery scheme can be used to prevent future errors in diagnostics risk management.

Figure 1 shows how failure event A can lead to a more serious failure event B. For example, failure event A might be a noisy signal response that leads to failure event B, an erroneous assay result. Event B could lead to an incorrect medical decision and patient harm, all based on the noisy signal response. While IVD manufacturers can always try to eliminate the possibility of failure event A, this is not always possible or practical. In most situations, detection and recovery can prevent further damage from failure event A. In this case, detection is achieved by an algorithm that examines the signal response, and recovery is achieved by suppressing the reporting of the result. Although detection is covered in risk management guidance, recovery is often omitted.

Recovery is particularly important when detection and recovery occur in separate organizations. For the IVD industry, ISO 14971 suggests stating a limitation in the package insert as a means of risk reduction, albeit the least desirable one. For example, a list of possible interferences is often provided in the package insert. This list could mean that the manufacturer has been unable to prevent such substances in a high enough concentration from causing erroneous results. The manufacturer has listed the substances in the package insert as a form of client notification. It is the clinical laboratory’s responsibility to perform a recovery, i.e., determine the concentration of interfering substances in every specimen. Unfortunately, this recovery step is expensive, so often no recovery is performed at all and erroneous assay results may occur.

Tip 5. Use Pareto Analysis

During product development, there are numerous observed failures ranging from cosmetic to patient safety issues. Moreover, every organization is faced with limited resources to address such issues and must optimally use their resources by asking which issues need to be addressed first and which can be deferred. Although not mentioned in ISO 14971, Pareto analysis is a common way IVD companies can answer that question.

Pareto analysis relies on classifying each failure event by its severity and the probability (FMEA, FTA) or frequency of its occurrence (FRACAS). Using FRACAS as an example, IVD manufacturers might multiply severity by frequency of occurrence (FO), which gives a quantity that can be sorted into a Pareto table or chart. However, companies need to be cautious when performing Pareto analysis. Some IVD manufacturers might illogically conclude that a high severity multiplied by a low FO has the same rank as a low severity multiplied by a high FO because they have the same numerical value. One way to avoid this mistake is to use weighted severities (e.g., avoid using the linear 1, 2, 3 scale).¹⁰ Another option is to use a nested Pareto, in which all events with the highest severity are sorted by descending FO, followed by the next-highest severity.¹¹

Tip 6. Validate Design with FRACAS

FMEA and FTA rely on the development team to think of all potential failure modes, which is a difficult task based on the list of recalls. Fortunately, observing failures through FRACAS does not require such knowledge. The FRACAS process is particularly helpful with software validation, which is often a large part of many IVD devices. Software verification and validation are usually conducted by a dedicated group using test scripts and other techniques. However, it is difficult to quantify the assertion that the software failure risk is acceptable through this type of testing because it is uncertain if every necessary test has been attempted. A close examination of recalls due to software failure often reveals that the requirements have failed, not the code, but the majority of software verification concentrates on test scripts (i.e., testing whether the code meets requirements).

FRACAS is helpful in validating software in two ways. The first way involves running the system as a user, which allows user errors to surface. Since the graphical user interface is the primary method for interacting with a system, user testing allows failures due to incorrect or insufficient requirements observed during analysis. In the second way, the developers can measure actual error rates with FRACAS and compare with a goal. FRACAS allows developers to estimate error rates in software testing using scripts through software reliability growth modeling.

Tip 7. Add Fault Trees to FMEAs

An FMEA is a table of potential failure events and, by definition, is limited to a cause and effect as well as the failure. Unfortunately, as described in tip 4, effects are often part of a failure event cascade. While it is possible to enter the appropriate failure effect in a FMEA table, it can be challenging because the effect may or may not be the next event in the cascade. Another problem is that FMEAs do not account for failure modes having multiple causes.

Figure 2. (click to enlarge) An FTA fragment.

FMEA is a bottom-up technique. An IVD manufacturer starts with a component, subsystem, or process step, and asks how this component or process step could fail. Such questions are useful, but should be complemented by a fault tree. A fault tree is a top-down method that starts with the most undesirable event and seeks causes for it. Figure 2 and Table IV show a fault tree and FMEA for the same failure sequence.

Table IV. (click to enlarge) An FMEA for the same FTA fragment in Figure 2.

Each FMEA and FTA event has the same information, although FTA will allow users to visualize products better. In Figure 2, in addition to the usual fault tree symbols, a dynamic gate (PAND) is used to show that the events in this gate occur in a sequence. Ideally, one software program should perform all FTA and FMEA steps to ensure that events in FTA and FMEA are the same throughout.

Tip 8. Combine Process Mapping with Hardware Analysis

Figure 3. (click to enlarge) An FTA of a failure with multiple causes.

An IVD device often incorporates hardware mechanisms that necessitate product risk management, including FMEA or FTA of hardware subsystems or components. In addition to the hardware analysis, several development processes are involved, including the design and manufacturing processes, and the actual process by which the product is used. Failures often have multiple causes resulting from interactions between the product and process. For example, in Figure 3, a fault tree view of a home glucose meter failure (which led to a recall) shows the interaction between the user and a software fault.

One FMEA pitfall is performing separate FMEAs according to categories, such as design, manufacturer, user, etc. This makes it more difficult to see multiple, interrelated failure causes. For example, in a user category-based FMEA, an IVD manufacturer might see each control measure listed as better training. A user performing the wrong step is cause for marking a failure’s control measure as better training. But the control measure could also be a design change, which reduces the likelihood of the user performing the wrong step. This ambiguity can be minimized by not performing separate FMEAs.

Tip 9. Understand Cognitive and Noncognitive Errors

Human error is a factor in every observed failure (i.e., a hardware failure is caused by a design or manufacturing error, ultimately made by a person). One of the goals of risk management is to ensure that sufficient control measures have been provided, which is where cognitive and noncognitive errors help.¹²

A cognitive error is defined as an incorrect choice due to insufficient knowledge, such as misinterpreting the necessity of setting a clinic’s product requirements or applying the wrong detection algorithm to prevent a hardware failure. Such failures are classified as being preventable, in which the suggested control measure is some form of training that ensures the correct information is shared with the project team.

A noncognitive error is an inadvertent or unconscious lapse in expected behavior. For example, a software verification step in which an employee proofreading the accuracy of 500 coefficients may inadvertently fail to detect a wrong value. Such failures are considered nonpreventable. With noncognitive errors, IVD manufacturers should focus on detection—recovery control measures, perhaps with a third-party employee double-checking the information serving as a typical detection control measure.

At-risk behavior is an action that increases risk when the possibility of a hazard is not recognized or is mistakenly considered to be justified. Reckless behavior is an action that consciously disregards a substantial and unjustifiable risk.

IVD manufacturers need to take into account all cognitive and noncognitive errors and all at-risk and reckless behavior for their processes, especially noncognitive errors by IVD device users. Users will make noncognitive errors despite proper instruction, and the IVD devices must account for such a possibility.

Tip 10. Continue Risk Management after the Product’s Release

In FRACAS, events are observed before the product’s release, while complaints are reported after release and are a subset of events observed prior to the release. In

Figure 4. (click to enlarge) FRACAS and CAPA methods play a similar, but separate, role in events, failures, and complaints.

FRACAS, each event is examined and classified as either “no problem” or an event that could lead to a failure. However, not all failures will become official complaints. Figure 4 illustrates how the FRACAS and CAPA methods handle such cases before and after the product release.

A FRACAS effort is often conducted by research and development, and complaints are often part of a formal CAPA process required by regulatory bodies. One subtle difference between FRACAS and CAPA is that FRACAS is intended solely to influence the design, while CAPA is conducted by IVD manufacturers to maintain customer satisfaction, although this can also influence the design.

Some IVD manufacturers may wonder why to even worry about the design after the product’s release, especially if the design or software has been labeled as frozen during the later stages of product development. The design is frozen only if there are no more changes, and the product is no longer supported. Manufacturers should view product development as a process that continues after the product release, with the benefit that IVD manufacturers that continue FRACAS after the product’s release lower the risk of recalls

---