Medical Device Link.

Weinger

Weinger: For more than 20 years, there has been a very close relationship between standards-making activities—both national and international—and regulatory activities by FDA. The AAMI human factors guidelines were first approved in 1988, but there was a period of approximately five years of activity prior to that approval. The guidelines were revised over the five-year period after 1988 by a committee that included several members of the present panel. The main changes reflected in the current version—which is designated HE48 1993 and is an AAMI/ANSI standard—are the inclusion of human-computer interaction guidance with regard to designing microprocessor-controlled devices, and the inclusion of a brief description of the human factors engineering process. Beginning in 1996, the committee began deliberating on how to revise the 1993 document, which was still insufficiently specific to medical industry needs and had some gaps because of evolving technology.

However, a larger concern of the post-1996 committee was that the general guidance about how one should design a display, for example, was perhaps less important than a broader approach that would define an optimal human factors design process—especially since this process could vary from device to device. So the committee began work on a separate document that was intended to be a national standard for the human factors design process. That document is currently out for final balloting, and is expected to be approved by July 1, 2001. Over the last year and a half, the committee has concurrently begun a parallel effort to revise and expand the classical "how-to" human factors design guidelines that are in HE48. Our current plan is to do this as a Web site, and in fact we very much need manufacturer support and encouragement to be able to deliver on this opportunity to provide good user interface design guidance.

I should have mentioned earlier that the AAMI human factors committee includes, in almost equal numbers, representatives from industry, clinicians, and others with interest or expertise in human factors and medical device design, including FDA representatives and human factors consultants.

Wiklund: Could you clarify one of the comments you made, which was that the last version of the standard was still considered less than ideal in terms of how well it is suited to medical devices? Perhaps you could retrace the history of why, even after a second revision, it still might not speak to medical devices as effectively as we'd like it to.

Weinger: I think the big problem, frankly, is that especially back in 1993—and even now—we just didn't have a lot of medical device–specific design guidance available. Even though individual device companies have a lot of knowledge and experience (although how much of it was documented is hard to say), this information was not public. The material that was available to the committee prior to the 1993 document consisted primarily of published standards from other domains, especially the military. And so the committee lifted heavily from military design guidance standards, and, together with some material from the NRC and other published design documents, tried to modify them as best they could for the medical industry. However, it was recognized at the time—and certainly since—that some of those modifications were more successful than others. In particular, the human-computer interface sections were not modified sufficiently to address the needs of medical device manufacturers with respect to building actual devices, particularly for critical-care applications.

MD&DI: Why did the committee initially focus on design process issues rather than principles of good design? What are the core elements of AAMI's recommended process?

Wiklund

Wiklund: What I can perhaps do is personalize the question. As a consultant, I'm asked from time to time to take a look at a medical device and render a judgment regarding its usability and safety by virtue of the quality of its user interface. Now, that's a really hard job to do. Occasionally there might be some low-lying fruit that you can pick—that is, obvious cosmetic shortcomings in the design, the nature of the labeling, the size of characters in a display, things that you can readily identify by inspection and application of known principles of good design. However, it's generally very difficult to make more overarching judgments about the usability of a device just by looking at it. Moreover, as a designer, you may come up with a design that you think would pass any litmus test in terms of good design practices, but, until you validate it in the context of a usability test, you can't be sure that you have developed a good design.

The AAMI work and FDA's guidance do stress good design processes, because we would all probably agree that such an emphasis represents our best hope for producing a high-quality user interface. As a practical objective, it's difficult to create perfection right off the bat by virtue of outstanding design talent being brought to a task. A more workable objective is to create an iterative process of researching users' needs and preferences, turning those into design goals, developing a design concept that reflects those goals as well as good design practices, then going ahead and modeling a design and having people interact with it and seeing how things go—and then repeating the process. You might liken it to the way you wash your hair: you wet, lather, rinse, and repeat. One could say the same thing about cleansing a medical device design of any kind of human factors shortcoming. The objective is to get end-users involved in the process of expressing their needs and preferences up front, and then evaluating the product by having them put their hands and minds on it and seeing how things go when they try to perform tasks. That's probably the most reliable way of producing a design that will perform well.

Hasler: One of the reasons we really focus on design process issues is that—despite the fact that people often want to just pick up some sort of a generic "cookbook" document and look for the exact "recipe" they want—there's simply too much variation in the user interface of medical devices for an encyclopedic guidance to work. In other words, if we're focusing on a ventilator or an infusion device, we could come up with great guidelines, but when it's a matter of the whole industry, an approach emphasizing good processes is initially the best way to proceed.

MD&DI: What is the status of international efforts to promote human factors in medical device design?

Carstensen

Carstensen: The International Electrotechnical Commission (IEC) is in the process of updating its big document, IEC 60601-1, which covers general requirements for the safety of electromedical equipment. As part of that undertaking, about a year ago they initiated efforts to develop a new collateral standard that, once in place, will become part of IEC 60601-1. Its number is IEC 60601-6; there are five other collateral standards, covering areas such as EMC testing. The first committee draft (CD) is scheduled for distribution to national committees for hearing and comment in February 2001, although the target date for publishing the final document is not until the fall of 2004. That sounds like a long time, but the good news is that, historically, companies will get wind of what's going on and procure copies of the first and second CDs and respond to them. They'll react in anticipation of a standard coming down the track, and derive much of the good effects well before the publication date of 2004.

The international standard—basically an international human factors engineering guideline—is based on the AAMI documents. But the IEC document itself probably doesn't occupy more than about 10 pages, plus an informative annex that tells you how to go about doing the job. That informative annex will be the newly revised AAMI human factors design process standard. The intent is to achieve global harmonization at the outset, as opposed to what we usually do, which is to have an international version of a standard and various domestic standards, and then get together years later and sort of argue about the differences and try to settle on something that is reasonably harmonious. This time, we're making sure it's harmonious from the beginning.

In addition, ISO Technical Committee 210 on quality systems has expressed an interest in joining the IEC working group that's developing the international version of the AAMI standard, so as to put out a joint ISO/IEC version of the standard. What that would do is allow us to expand the scope beyond electromedical devices to include all medical equipment.

MD&DI: Rod, as a human factors specialist working at a large company, what is your view of the new regulations? What are the greatest challenges you face in responding to them? What about cost pressures? Finally, are there differences in the way the regulations affect how you develop products for domestic versus foreign markets?

Hasler

Hasler: Regarding human factors concerns in the medical device industry, companies can be divided into two camps. The camp I come from recognizes the importance of human factors at least as long ago as the early 1990s. At that time, we implemented a customer-focused process to define an IV infusion system, and quickly discovered that the feature most desired by the customer was ease of use. This drove us to see the importance of hu-man factors practices, and how good human factors could benefit us. So my introduction to human factors was really on the marketing side—how to make better-selling products that are easy for customers to use.

And I think that's probably where you're seeing the companies that jumped into the discipline of human factors early on—they were really utilizing it for the ease of use, and to drive a better product to the customer. Those who didn't recognize that are a little bit behind as far as converting.

I think that most of the larger companies followed this same route, and typically have long-standing human factors programs. Many smaller companies are still in something of a catch-up mode; they're trying to understand exactly what is required and how they can implement it. But I believe that this regulation has a very strong upside for the entire medical device industry—it will really improve the industry as a whole, as far as reducing design errors.

MD&DI: Do you notice any differences in the emphasis on human factors in products destined for the U.S. market versus the overseas market?

Hasler: What you see much less of in Europe is human factors used as part of a marketing strategy or approach. Regarding domestic and foreign markets, however, one of the problems in the device industry is that we tend to develop products that are oriented toward and designed for a specific customer in a specific country, but are then released to other countries with no changes. Whereas even though the actual clinical application may remain the same, there are often differences on a country-by-country basis in how the users react to the design and employ the device. So the biggest concern I have on a global front concerns releasing a product that was designed for one market into multiple markets.

MD&DI: How do the new rules affect the marketing of devices that were developed before the human factors regulations were adopted? What happens with products that represent slight modifications to older products that may not have incorporated good human factors design principles?

Wiklund: I'd be happy to answer this question from a consultant's point of view. I think this is going to be a great source of anxiety in the future for those companies that are not getting clear signals as to their vulnerability in terms of selling any product that hasn't benefited from a good human factors design process. Because many companies will be introducing products that represent slight variations of previous versions, one could argue that they should conduct a thorough human factors evaluation of the modified design—which might infer getting customer feedback in the context of usability tests, and so forth. Of course, a company that has marketed a product for a long time—a product that was originally approved by FDA—might ask itself whether it really needs to go through all of that effort after making only minor changes to the product.

Compounding this whole issue is the fact that, once regulations are in place and people become more aware of them, a company that doesn't follow good human factors processes could be accused of not applying due diligence in pursuing state-of-the-art practice. In other words, you raise the possibility of legal liability exposure if a firm fails to follow the new standard. My guess is that companies that carry out minor modifications to existing products will at least want to conduct a usability test to confirm that the new changes are in fact good changes. They'll want to make sure that they haven't inadvertently introduced other kinds of problems, or somehow corrupted the preexisting design in a way that could lead to user error. So at a minimum, I think that companies taking a conservative and careful approach will likely begin doing more usability testing than they would have before these regulations were in place.

Hasler: What I'm seeing in the industry reflects very much what Mike has just described. It can be quite confusing deciding how to handle a product that you've been producing for, say, 20 years once you recognize that a minor feature needs to be changed. According to all previous methods of evaluation, you would have gone ahead and made the change for any additional units sold. Now, however, once you start dealing, for example, with the user interface, you may realize that other aspects of the device may not meet the present standard and may need to be changed. Although a company certainly wants to support its customers, you're now faced with a full-blown project—a whole rework—of something that really isn't the product line you want to move into the future. In short, it is very difficult to understand how the regulations should be applied to older products.

MD&DI: What are the views of FDA on this issue?

Sawyer

Sawyer: Any changes being made in product design should be run through a company's design control procedures. This applies to changes during the development of a brand-new product or modifications to an existing one. Regarding human factors, the question of existing devices is one that manufacturers occasionally have asked us about. In the case of so-called evolutionary product development, many small modifications may have been made over an extended period, and there seem to be two questions: How big a change is necessary before human factors evaluations are required, and how do you account for every design change made over a number of years? First, since small changes in hardware or software design can affect the way a device is used, an analysis should be done to decide whether a change is trivial with regard to the user interface. Depending on the nature of the change, a human factors test may be necessary, as Mike suggested. Second, we realize that, 5 or 10 years ago, many manufacturers were not doing substantial human factors evaluations, and there may be no data on early design modifications, or even on the original design. But in making a change today, a company certainly can examine its complaint files, which provide a kind of track record, or baseline, with respect to safety.

MD&DI: What is the function of the design history file as it relates to the new regulations? Can you give a quick overview of what FDA is looking for in terms of documentation?

Sawyer: A design history file can exist in various formats, as long as the company can produce it for our field investigators. If you look at our regulation, several areas are stressed regarding what is documented. Design inputs, or requirements, are important, and especially relevant to human factors are descriptions of the user population, working environment, potential hazards, and basic design concept. Verificaiton data will pertain to inspecting, analyzing, and testing a device against such requirements. In human factors, this can involve anything from paper exercises like task analyses and walk-throughs to full usability tests with, for example, computer simulations. Validation addresses a broader question—Will the device work in a real-world situation? Human factors validation testing is usually done on production-level models. The tests are simulations, and operating conditions should be as realistic as possible. On the other hand, although clinical trials sometimes yield some useful human factors information, because they involve actual patients, highly trained device users, and substantial manufacturer support, they often don't produce good and representative human factors data, and there are obvious limitations with regard to patient safety. In any case, as with any other design discipline, human factors efforts should be well-documented.

Wiklund: In addition to some of the analytical results just mentioned, I would think that the design history file would include some of the classic products of human factors studies, such as reports from user focus groups, summaries of conceptual inquiries, on-site interviews with users, and so on. It might include a usability test plan and a report of usability test findings. If we back up and talk about actually producing a design, the design history file might include usability goal statements—the goals created to underscore the design effort and keep the design team focused on what it wants to accomplish.

Sawyer: I agree, and this raises an important issue. Information from early usability studies and concept stages that may precede the actual formalization of the project are not required by FDA. However, it makes a lot of sense and can work to a company's advantage to include data from preproject or predevelopment efforts.

Wiklund: My recommendation to companies is to take some of their market research resources and create a close collaboration with human factors professionals, so that some of these traditional research activities can be given a new spin or expanded to address some of the human factors issues that concern FDA. Many companies do a lot of this kind of work, but perhaps without the heavy human factors focus that is now warranted. So for many, it may not be a great cultural change in how they approach things, but rather simply an expansion and full embracing of the human factors requirements.

MD&DI: For companies seeking assistance, where can they find human factors specialists?

Wiklund: It's a great time to be in the human factors business, predominantly because the dot-com sector has soaked up a lot of human factors specialists. Human factors professionals are trained in graduate programs at many institutions around the country, so by contacting the Human Factors and Ergonomics Society, you can get a listing of where these people are training, and potentially recruit people from there.

In terms of finding human factors specialists who have more experience in the medical arena, that's not a huge population—though I expect many more people to gravitate toward the medical industry as it embraces human factors and its needs grow. I don't have a good road map for finding experienced people. Generally, they're working in companies and participating in the medical systems section of the Human Factors Society. You can obviously contact human factors consultants. You can also turn to FDA, I understand, and they'll help you in terms of explaining what their goals and objectives are regarding the regulations.

But in terms of getting people to do the hands-on work, many companies will have to draw on the capabilities of people who are perhaps peripheral to the human factors field but are ready to invest themselves more substantially in practicing human factors. This might include personnel involved in tech writing; it might be those involved in industrial design–related activities; it might be people leading software development efforts. These are the folks who are going to have to become students of the human factors process and design principles. I don't think anybody has the patent rights—whether they're formally trained or not—to doing good human factors work. Obviously, I'm biased toward thinking that human factors is a profession that should be practiced by professionals. But recognizing that there will be a shortage, my sense is that people who may not be formally trained in human factors but are experienced in related disciplines should be able to do a reasonable job and make a difference.

MD&DI: Do companies typically think they need to hire human factors specialists to address these questions, or are they confident that individuals with different backgrounds can fulfill some of the functions we've mentioned?

Hasler: I think that both types of individuals add value to a product development process. Both have their pros and cons. Typically, a person who has considerable time with a company has some clinical experience and a solid understanding of the product being developed and of the needs of its users. I would agree that it may not be critical to have a degree in human factors, but I think that some formal training or course work should be required. There is certainly some excellent literature available for the study of human factors. However, some people look at it and say, "Well, that's common sense," and my experience over the years has been that "common sense" is not as common sensical as you might wish. So, I would definitely recommend some training.

Smaller companies or start-ups can do supplemental training or hire a consultant to review their work, perhaps designating an individual to be the human factors "point person" in working with the consultant. The key for the company is to get the human factors input early. Too many firms find themselves with a product at the very end of the design process, and they're asking themselves not is it okay from a human factors point of view, but is it acceptable? The time to apply human factors is at the very beginning, almost before the engineers put anything on paper.

Wiklund: There's a lot of evidence showing that any amount of money spent on human factors—especially early in the process—pays major dividends down the line. Books have been written that track the payoff from usability investment as ranging anywhere from a 3:1 return to upwards of 100:1. The benefits can include avoiding future liability claims and getting the product to market faster, since you don't encounter usability problems late in development and because the product will likely proceed through the regulatory process more smoothly if it reflects good human factors. Among other benefits are reduced customer service expenses and more-effective sales presentations leading to improved sales. So in addition to being good professional practice, a company's investment in human factors—whether it be hiring a person in-house or using consultants—is going to pay off for them, in my opinion.

Hasler: I agree completely. First, your product development process is smoother and faster. Second, good human factors can be marketed and sold, because ease of use really shines when customers take the product in their hands. They might not even know just why a well-designed product is so special, but they realize it's special. Finally, those flaws that you caught early on will not be showing up in the field and you won't have field actions or potential product recalls to deal with.

MD&DI: Does FDA believe that the new regulations are having their intended effect? Are they, in essence, paying off? Do you have any advice for our readers and the industry on the importance of good design in medical devices?

Sawyer: If you're talking about clear-cut evidence, it's a difficult call, in that the design controls really haven't been in effect for very long. Probably the most concrete evidence that we see is in premarket submissions when we consult our Office of Device Evaluation, and indeed we've seen some products that have had some really excellent design efforts behind them. And although it's not typical, I saw one product for which the company claimed to have put a million dollars into their human factors design process. I don't know whether it's true or not, but they certainly had a very interesting, well-designed device and they had done a lot of work on it.

So certainly there are some companies that are paying more attention to human factors, and of course I've already talked about the study we've done and the consultants who are finding more and more business out there. Overall, however, there's still an awful lot of variability. We see submissions with very little evidence of human factors work, and it shows up in the design. And then there are many cases that are sort of in between—when a manufacturer submits a product for approval saying, "Well, we tried to do a little human factors but can you tell us more," and so on, which initiates an ongoing dialog. In general, I think we're optimistic, though the variability remains and it's difficult to make simple before-and-after comparisons at this point in time.

Carstensen: What we're seeing is certainly encouraging, even though we only get a limited picture. However, taken as a whole, the industry has a ways to go. Our mission is to help firms get there as quickly and as painlessly as possible, so they and their customers can enjoy the benefits of good human factors design.

In a sense, our task should be an easy one. If we can convince the industry that competent human factors efforts pay off in terms of bottom-line business, the whole idea should sell itself, and that's our goal. We're going to continue to educate, to get the message out, and to persuade the industry that this is in its own self-interest. I firmly believe that such an attitude will be more effective than the alternative—to sit here with the big FDA stick threatening to beat industry over the head if it fails to "comply" with the regulations. That kind of approach merely results in companies putting window dressing on things and trying to convince FDA that they are doing a good job with human factors when in fact they're doing a superficial job. We want to get them to do it for real, so it does pay off, so it truly is effective.

MD&DI: Do you think that message is resonating with
the industry?

Carstensen: We hope it is. To the extent that we speak to the industry—through articles and conference presentations and so on—we always try to sell the benefits of a strong human factors process on its own merits. But the impact of that message can be hard for us to judge. Have the captains of industry, the people really making the management decisions, really gotten that message? I don't know. There's no easy way for us to measure that.

Hasler: From my vantage point, the message is getting out there. It isn't moving en masse just yet, but it definitely is gaining momentum, and I can't imagine what would slow it down.

Carstensen: We're all probably seeing about the same effect.

Wiklund: It's obviously very, very difficult to count nonadverse events. The sum of our efforts in good human factors engineering is to avoid what we could refer to as deleterious outcomes: if we do our job well, we'll see people using products on a daily basis without mishap. It may not be something that's going to grab headlines—for example, that so many infusion pumps were used effectively on February 1 of the year 2001. But given the great deal of alarm right now about the amount of user error related to medical devices and its effect on healthcare, there's certainly going to be considerable attention paid to this whole issue, and a full embracing of human factors will start to be a big part of the solution. Once we succeed in reducing user error and start achieving day after day of good outcomes, at that point—like airplanes getting safely where they need to go—there may be no news, but no news will be good news.

On the notion of human factors paying off, I think it's certainly paying off for the companies that have invested. The companies that have invested in good human factors are doing a pretty good job of burying the competition. It was mentioned earlier that studies demonstrate the fact that usability tends to float to the top in terms of a design priority. That's true of all the research we've done regarding dozens of different kinds of medical devices. Usability is always rated very high, if not the highest, among the 10 or 12 most important attributes. So the marketplace will have a strong effect in reinforcing good human factors practice. Speaking as a person who has participated on a jury for the Medical Design Excellence Awards, I've noted that some of the best products nominated for awards make a big case about the quality of their human factors design efforts—really showcasing that as evidence that they in fact do have a design that warrants recognition. This is one more indication that an awareness of the importance of human factors is really taking hold.