IVD Technology Magazine
IVDT Article Index

Originally published January, 1998

Commentary

All that glitters is not a gold standard

Roger Briden

At FDA, the performance of microbiological IVD products has traditionally been measured against "gold standards" made up of routine biological cultures. Although a manufacturer's premarket notification (510(k)) may claim substantial equivalence to any number of predicate devices, when it comes to demonstrating performance characteristics FDA expects these to be measured relative to such gold standards.

A problem with this long-standing practice is that biological cultures are subject to the influence of a wide range of variables that make their use as gold standards somewhat problematic. To begin with, laboratories use a wide variety of growth media, any of which can affect organism recovery. Culture methods are useful only for viable organisms, the growth of which can be influenced by such parameters as the initial organism load, the strain of organism, incubation, atmospheric conditions, and antibiotics or over-the-counter preparations used by patients. Sample handling, storage, and transport can also effect culture results. Above all, the successful culturing of any organism relies heavily on the technique of the laboratorian, thus opening the door to wide variations in the quality of culture results.

For all of these reasons, it is difficult to establish definitively how an assay performs relative to a "routine" biological culture. The outcome of such an effort depends on the quality of the culture result. Moreover, advanced technologies whose performance equals or exceeds that of routine cultures present problems for both FDA and manufacturers, including how to compare the performance of such technologies against the standard, and how to present the relevant data.

Last October, Kimber Richter, deputy director of FDA's Office of Device Evaluation (ODE), announced that the agency intends to hold a panel meeting early this year to discuss its use of gold standards. It is encouraging that ODE should recognize the potential problems with culture-based gold standards in this way. Together with the agency's efforts at reengineering and modernization, hopefully we will see appropriate changes in what FDA uses as a gold standard, and how it does so.

Certainly, a part of this panel discussion should center on the technical difficulties inherent in using biological cultures as gold standards. But some part of the meeting should also focus on FDA's overall policies for the use of such standards and how the agency uses--or should use--medical practice guidelines in deciding what restrictions to require in product labeling. The panel should also discuss how far FDA should be permitted to dictate, directly or indirectly, the practitioner's use of an assay. The need for a discussion of these issues can be demonstrated by examining current FDA policy and practice regarding rapid assays for group A strep and group B strep.

In the case of strep A, FDA's policy, based on a recommendation from the American Academy of Pediatrics' Red Book, is that the results of all rapid direct-antigen assays must be backed up by comparison to culture. In adopting such a policy, the agency seems unmoved by the fact that organizations issuing such recommendations are traditionally slow to respond to advances in technology. Moreover, such guidances are typically written with patient management in mind and are not intended to be used as part of a substantial equivalence (510(k)) review.

It is problematic when FDA misunderstands the viewpoint of a guideline-writing organization, or misapplies guidelines intended to advise practitioners. One result can be agency requirements for product labeling statements that restrict practitioners' use of an assay and amount to an indirect regulation of the practice of medicine--an area FDA professes to stay out of. Similarly, the agency's blanket application of recommendations written by external groups that don't keep up with advancing technology hinders both industry and the practitioner.

FDA's policy does not allow manufacturers to claim superiority to a gold standard even when the company's data would support such a claim. Nevertheless, the agency does permit manufacturers to display data demonstrating such performance. Such a policy appears to be a mixture of not disagreeing with the data combined with retaining the sanctity of an old standard. But the policy presents a problem when new technology produces assays with sensitivity equal to or better than the standard. It allows poor performers to look good, and prevents better performers from standing out. And it is difficult to ensure uniform compliance by all manufacturers.

Medical practitioners encounter these problems in the product labeling that requires them to perform culture back-up in spite of product data that would not logically require it. Certainly, it makes little sense to universally back up a more-sensitive test by using a less-sensitive one. And recent studies looking at medical practice related to the diagnosis and treatment of strep A infection have demonstrated that patient management is not compromised when a rapid assay of adequate sensitivity and specificity is not backed up by culture.

Although FDA may intend that practitioners should exercise their own judgment about the use of such diagnostics, the labeling that the agency requires leaves little latitude for such professionals. In a litigious society such as ours, any practitioner who chooses to deviate from the instructions provided in product labeling runs a high risk of liability for his or her actions.

In 1996, FDA issued a safety alert to medical practitioners about the proper use and interpretation of direct antigen assays for group B strep in testing pregnant women and infant urine. The purpose of the alert was to warn practitioners that some of the group B strep (GBS) tests on the market had unacceptably low sensitivity, and that the agency believed more data were needed to show a correlation between antigen in infant urine and GBS treatment.

The agency subsequently informed all manufacturers of strep B tests that if they intended to keep their products on the market, they would have to submit performance data comparing the products to a new, more-sensitive standard: broth culture. Several manufacturers withdrew from the market on their own initiative, and many practitioners interpreted these actions to mean that FDA would soon withdraw all such tests from the market. In clarification, Steve Gutman, director of the ODE Division of Clinical Laboratory Devices, told industry representatives that the agency did not intend to remove products from the market as long as data relative to the new standard were provided. Pending receipt of those data, products that performed according to their stated specifications could continue to be used in accordance with their labeling.

FDA's alert tarred all rapid strep B assays with the same brush, implying that none had sufficient sensitivity to be clinically effective. Industry was never given a clear opportunity to demonstrate otherwise. Many published and soon-to-be-published studies have concluded that rapid assays of appropriate sensitivity are very useful in the management of GBS disease. Those manufacturers whose products remain on the market continue to work with FDA to define and establish what is necessary to address the agency's concerns.

Among practitioners, however, there remains considerable confusion about the intent of the safety alert. Some continue to wonder if all strep B assays will be withdrawn from the market, and many are concerned that FDA's actions have created a new and unanticipated liability dilemma. Just as in the case of strep A, FDA's strep B alert defines a standard for use that removes much of the practitioners' decision-making ability and places them in the position of using an assay in a manner inconsistent with its labeling.

And once again, as in the case of strep A, FDA's handling of strep B assays raises questions about what aspects of a product the agency should consider as part of its product clearance process. The agency should review its use of external guidelines. Contrary to the impression one would get from reading the GBS management guidelines issued by the Centers for Disease Control and Prevention, there is more than one way to effectively manage GBS disease. New studies continue to demonstrate that there are efficacious management alternatives that make use of a judicious combination of rapid assays, risk factors, and culture.

One can ask whether FDA's authority over product approvals should be permitted to cause such a cascade of problems for manufacturers and practitioners. Perhaps FDA's policy was valid for older technologies, but it now needs to be changed. The agency would do better to restrict its labeling requirements to factual matters related to a product's performance. As long as these facts are presented accurately, the agency should not dictate further the clinical use of the product. Such an approach would allow practitioners to evaluate which products provide the performance needed for the diagnosis and treatment of diseases.

With all the changes FDA is undergoing, it is heartening to hear that the agency is looking into how best to ensure realistic and uniform assessment of a product's performance. Hopefully such needed changes will be carried out in a timely fashion.

Roger Briden is director of regulatory affairs at Biostar Inc. (Boulder, CO).