Medical Device Link.

Originally Published IVD Technology November/December 2003

IN PERSON

A workshop sponsored by NIST provides IVD manufacturers with information on calculating uncertainty of their calibrators.

Marc Salit, PhD, is research chemist in the Chemical Science and Technology Laboratory at the National Institute of Standards and Technology (Gaithersburg, MD). He can be contacted via e-mail at salit@nist.gov.

The traceability requirement of the IVD Directive has presented many IVD companies with a challenging obstacle to navigate. The National Institute of Standards and Technology (NIST; Gaithersburg, MD) has stepped forward to provide IVD manufacturers with assistance on meeting this requirement. Whereas the calculation of uncertainty of calibrators has always been part of the research and development process for IVDs, these calculations must now be presented as part of a device's product literature. One of the implications of this development is that users will begin to place an even higher importance on their impressions of these calculations, thereby having an effect on the success of a product in the marketplace.

In order to shed some light on how manufacturers should go about approaching the calculation of uncertainties for their devices, NIST recently held a workshop for the IVD manufacturing community. IVD Technology editor Richard Park spoke with Marc Salit, PhD, research chemist in the analytical chemistry division at NIST, to explore the goals of this workshop and the role NIST will play in providing further help to IVD manufacturers for the calculation of the uncertainty of their calibrators. In this interview, Salit discusses the origins of and basic concepts discussed in the workshop, as well as the role NIST hopes to play in supporting the IVD community in the future.

IVD Technology: Please provide some general background information on the workshop titled "Practical Guidance for the Calculation of the Uncertainty of Calibrator Values for IVD Manufacturers" that was sponsored a few months ago by the Chemical Science and Technology Laboratory (CSTL; Gaithersburg, MD) at NIST.

Marc Salit: Following the creation of the European Union's IVD Directive a few years ago, it became clear that there would be a need for supporting metrology. In other words, there was a need for some formal transfer of measurement science ideas and measurement science thinking into the IVD community. A workshop examining this role for measurement science was held at NIST in 2000. Workshop attendees explored the issues that would arise from the implementation of the IVD Directive.

One topic that clearly required some development was the treatment of uncertainty as applied to measurements that established the traceability of calibrators. This need for more development stemmed from the expectation that, ultimately, considerations of measurement uncertainty would propagate right down to the patient bedside and to all end-users of IVD tests.

In light of the fast-approaching December 2003 deadline for compliance with the IVD Directive, the IVD industry sought our expertise as gained from our experience in establishing uncertainty budgets for measurements—both chemical and clinical measurements—and we willingly agreed to contribute what we could to the community of IVD manufacturers.

Did NIST conclude on its own that there was a need for metrology, or did you identify this need in conjunction with the IVD industry?

The greatest pull came from the IVD industry asking for help in meeting the requirements of the IVD Directive. People were asking for help, particularly in meeting the requirements defined by Section A3 and Annex 1 of the directive, which states: "the traceability of values assigned to calibrators and/or control materials must be assured through available reference measurement procedures and/or reference materials of a higher order."

In order to meet this traceability requirement, manufacturers need to be able to estimate uncertainties. To demonstrate traceability, IVD manufacturers must have both a reference to compare with and a method of comparison. When that method of comparison is applied, there will be an uncertainty in that comparison, and that uncertainty needs to be propagated into the uncertainty results for the calibrator. Manufacturers required more assistance for calculating this uncertainty.

In order to understand how measurement uncertainty applies to chemical analysis, and in particular as it has been applied to measurements over the last couple of decades, we had to look at developments from the International Organization for Standardization (ISO) groups and committees that ultimately led to the publication of the Guide to the Estimation of Uncertainty in Measurement (GUM) in 1995.

The formalization of how to go about establishing uncertainty budgets has been applied across all sorts of measurements, from the international system of units (SI) like time, mass, length, and amount-of-substance, and to units that are not a part of the SI, like the International Units maintained by the World Health Organization (Geneva). The conceptual approach for estimating uncertainty is the same, regardless of what is being measured.

Using that formalized method of calculating uncertainty and taking into account the burgeoning adoption of accreditation to ensure consistent quality of laboratory results, people performing chemical measurements called for a method of establishing uncertainty budgets. An uncertainty budget is an accounting for all of the sources of variability in a measurement. Creating an uncertainty budget calls for laying out all of these sources quantitatively.

The need for a method to calculate uncertainty budgets was strong for the IVD industry because these calculations can be complex for IVDs. When making chemically based measurements there are often potential interferences, and complicated measurement models may be required to ultimately derive an answer.

The difficulty lay in how we were to account for the cumulative variability in all the steps taken to arrive at a diagnostic measurement. IVD manufacturers were also concerned with performing these uncertainty calculations in a practical fashion, without turning their laboratories into organizations dedicated to establishing uncertainties rather than doing chemical measurements. They also wanted to make sure that their uncertainty calculations would be accepted by the international community.

NIST and the Cooperation on International Traceability in Analytical Chemistry (CITAC; Geel, Belgium) have been involved with some ongoing work on the implementation of traceability for analytical chemistry. CITAC published a guide in the late 1990s called, Quantifying Uncertainty in Analytical Measurement (QUAM). And that has been a popular tool. There has been awareness among the IVD device makers that such a guide exists and that NIST was involved in its development. It is easy to use for someone doing chemical measurements, which ultimately is what IVD devices are doing.

Framework of the Workshop

How did NIST decide that presenting a workshop was the best way to address the needs of the IVD community?

The heightened need for understanding how to calculate uncertainty originates from the requirements of the IVD Directive. CSTL already had some of the built-in expertise based on its work with CITAC. Thus it was natural for CSTL to provide guidance on how the uncertainty of calibrators should be calculated.

This collaboration is a component of a larger partnership between NIST and the IVD industry in which we are working to develop the proper reference materials, methods, and other tools needed to establish the validity of devices as seen through the lens of the IVD Directive. The IVD Directive is, in fact, informed by the past decade or so of thinking that went into the development of ISO 17025 and the accreditation community's rise to the forefront in light of the ISO 9000 series, the quality assurance or quality system standards.

What were the goals of the workshop?

One goal of the workshop was to interactively communicate some of the concepts that would enable the attendees to calculate uncertainties of the measurements to assign values to their calibrators and internal controls. Another goal was to create a Web-based archive of the workshop for folks to be able to use as a reference as well as to reach a much wider audience.

The American Association for Clinical Chemistry (Washington, DC) industry division has requested that NIST hold additional workshops on uncertainty calculations in the future to provide an interactive venue in which to communicate this information. Until then, the Web-based archive will be a good resource for people to refer to when they are trying to get up to speed on meeting the requirements of the directive.

What were some of the highlights of the workshop?

The topics covered included a very cursory look at the fundamental statistics underlying uncertainty estimation from a fairly practical, pragmatic point of view. There was not a lot of theoretical discussion, but a framework was put together to present the underlying statistical principles used for uncertainty estimations.

Attendees received down-to-earth practical example-based instruction on doing uncertainty estimations. These examples were presented as spreadsheets, and people were able to watch the example build itself, step by step. Advanced topics were covered as well, including some of the more subtle issues of uncertainties arising from the use of calibration procedures.

We also presented the CITAC guide and how it could be used in a practical fashion to estimate uncertainties, both for developing a device and as a reference for doing uncertainties for calibrators and then establishing traceability to higher-order standards or higher-order methods. We intend to continue to use this guide. It has been very helpful. It is available via the Internet at www.measurementuncertainty.org.

Another highlight of the workshop that is described in the CITAC guide is a tool that provides an effective automated technique for calculating uncertainty. It is called the Kragten Method, and it is a spreadsheet-based technique. The Kragten Method is described in volume 119 of Analyst in an article titled "Calculating standard deviations and confidence intervals with a universally applicable spreadsheet technique."

Where Uncertainty Fits In

How would you define uncertainty, and how is uncertainty applied to IVDs?

When we make measurements, and an IVD is ultimately a device that makes a measurement, there is always some dispersion, some variability, in the results. Therefore, there is uncertainty associated with that reported result.

The magnitude of uncertainty is not a central factor. What is important is that there is always a dispersion of measured values about some central tendency or a mean. Calculating and considering this dispersion is important because ultimately when we are making measurements, we must compare the measurements against either an expectation or a limit.

End-users need to be able to determine whether useful measurements are higher or lower than a given limit. In order to make an informed clinical decision based upon technical results, they need to know what the expected dispersion of those results is. That is why uncertainty became bundled with the establishment of traceability.

Calculating uncertainty is important to IVD manufacturers because they are developing and producing machines that generate measurements. Ultimately, they are marketing the results and information generated by their devices. That information is only partially complete unless one can speculate about how those data are expected to vary, both as the result of the measurement and as a result of the analyte that is being measured.

How should IVD manufacturers apply uncertainty principles to processes involved in their quality control and quality assurance systems?

When developing an uncertainty budget, a scientist or engineer has obtained an understanding of where the sources of variability arise from in a measurement. That variability is a terrific measure of the performance of the manufacturing process.

When a process is in control, it will genereate a dispersion of results that are consistent with the uncertainty budget. When the process goes out of control it will start to show more variability, and dispersion coming in places that are unexpected. This dispersion is a great indicator of when a process needs management in order to bring it back in line. That is one way in which uncertainty calculations will affect product quality and manufacturability.

Ultimately, having an uncertainty associated with the measurements performed by a product, and being able to report that uncertainty to the end-user and any other customers, will let the customers make better use of the product and better judgments based on test results.

Before the traceability requirements of the IVD Directive, was there no real requirement for IVD manufacturers to calculate uncertainty for IVD products and devices here in the United States?

Every investigator who has put energy into developing new tests has considered whether their test will be useful, based upon an understanding of what the expected dispersion of the results would be. In other words, they have taken into consideration what uncertainty users would receive in their instruments' results. Calculating uncertainty has always been a natural part of doing quantitative science.

What's new with the IVD Directive is that people are being asked both to quantify uncertainties and to report them as part of the documentation associated with the product. However, measurement of uncertainties has been recognized as being an important component of performance all along.

Calculating Uncertainty

You mentioned that IVD manufacturers are faced with several hurdles and interferences when calculating uncertainty. Where do these difficulties come from?

Particularly for chemical measurements, uncertainty estimation can be complicated. For chemical analyses, there are a number of operations that create variability of results. Some of what complicates the uncertainty estimation for a chemical analysis is the sheer number of sources of variability. This multitude of elements has required some new thinking about how to evaluate the uncertainties of the results of a chemical test, as opposed to measurements that may have a more straightforward uncertainty budget such as physical measurements of mass, some of the electrical units, or time.

The current thinking in chemistry is to capture groups of uncertainty sources when validating methods. Results of method validation studies and proficiency testing activities are employed to estimate groups of uncertainty components in a more holistic method. This is one of the strengths of the second edition of the CITAC guide. This edition provides guidance via this holistic "top-down" approach, instead of using a bottom-up component-by-component approach as its tenet.

How are the uncertainty components grouped?

They are grouped together by measuring the variability of the chemical analysis. For instance, I may be establishing the uncertainty performance of a chemical measurement that may be influenced by both change in temperature and variability in the variety of volumetric flasks that could be used. In this case, I could make a large number of these measurements, maybe a dozen or 15 measurements, over the period of a week in a laboratory where the temperature is subject to natural variations. I could use four or five different volumetric flasks, representing the range of available flasks, throughout that week.

I can assess the variability in that population of measurements, and can conclude that I have sampled across the representative temperature range and representative population of volumetric glassware. Thus, the factors of temperature variability and variability between volumetric flasks would be grouped together.

What further developments in calculating uncertainty for IVDs will emerge in the future?

Several projects that are currently under way aim to automate the estimation of uncertainty and incorporate this automation into new products. Some of these projects have led to or will soon result in commercial products based on solid principles. These software packages and tools ease the labor involved in some of the calculations and the experiment design work required for establishing an uncertainty budget for a measurement.

In addition, some research activities will start to yield fruit in terms of software that guides experiment design to capture the variability of various measurements. Over the next few years those tools, which are in research labs right now, will become more routinely available.

Are IVD manufacturers involved in these automation projects?

I am not aware of anything being developed specifically for the IVD industry, but there are commercial activities that will ultimately yield products to do uncertainty budgets. In addition, the CITAC guide mentions an automation tool that is called the Kragten Method, which is quite useful.

The Kragten Method is a spreadsheet method for uncertainty calculations, which is quite easy to use when doing propagation of errors work. It is especially useful for exploring "what if" kinds of questions. For example, when trying to refine an IVD technology, one could use this method to determine how a final result would be affected if one component of a test was made more sensitive and less prone to variability.

Once an uncertainty budget has been established, processes can be refined so that they are more effective. Uncertainty budgets should enable manufacturers to invest in developing areas that will bring them the biggest bang for the buck, without spending time, energy, and resources on limiting variability in areas that do not contribute a significant source of variability to the end result. It is part of doing business with care.

Although they are not being developed exclusively for the IVD industry, is there any possibility that if they come to fruition, those uncertainty calculation products could have a collateral effect on the IVD industry?

Absolutely. If these projects do come to fruition, one could expect tools that would be directly applicable and useful to the IVD industry as well as those trying to establish uncertainty budgets for measurements there. Future Developments

How can IVD manufacturers contribute to the continued development of tools for calculating uncertainty in their products and devices?

The truth of the matter is that nothing associated with calculating uncertainty is new. None of the math is new. The newest advance in thinking came in the late 1980s and early 1990s when GUM was being developed. The novel contribution of GUM was to transform estimates of uncertainty components that were based on scientific judgment (rather than having been determined statistically) into standard deviations. These components can then be applied with rigor, regardless of the genesis of the uncertainty components.

Estimating and calculating uncertainty is a mature science. What IVD manufacturers can do is to provide tools for calculating and establishing uncertainties for their own measurements. They can integrate these tools with their IVD devices so that the end-users can calculate uncertainties in their results, thereby providing a superior product that will enable the end-user to make better decisions based upon test results.

How will CSTL and NIST build on the uncertainty workshop, and what further work does CSTL plan to do in the area of calculating uncertainty in IVDs?

CSTL will make the uncertainty workshop available in archive form. We will likely continue to carry the message to appropriate venues and reprise this workshop in an interactive fashion. These workshops are a key way in which we can engage with this critical community at a critical time for them, as they are being asked to provide evidence of traceability for the calibrators in their products.

We are partnering with NCCLS (Wayne, PA) and the International Federation of Clinical Chemistry and Laboratory Medicine (IFCC; Milan, Italy) in an effort to provide a guidance document on establishing traceability, of which uncertainty is a part, along with establishment of commutability. We are working on this guidance so that the IVD industry is able to flourish internationally and no technical barriers to trade prevent U.S. companies from being able to market their devices anywhere in the world.

NIST is playing a leading role in the Joint Committee on Traceability in Laboratory Medicine (JCTLM), which was created in the spring of 2002 by 60 high-profile stakeholders in a meeting held at the International Bureau of Weights and Measures, commonly referred to as BIPM (Sèvres, Cedex, France). NIST is cochairing the Working Group on Reference Materials and Reference Measurement Procedures. This working group is charged with establishing a process for identifying the higher-order certified reference materials and reference measurement procedures that will be required for IVD industry compliance with the IVD Directive. The higher-order standards identified through this process will ultimately be published in a database that will be maintained by BIPM and the IFCC, and will be publicly available.

I think it is important to understand that the work we did via the uncertainty workshop is in context with a multi- faceted effort to make sure that the IVD manufacturing industry is able to sell its products everywhere and allow manufacturers in the United States to produce the best possible products to bring to the market.

The archived Webcast of the CSTL workshop, "Practical Guidance for the Calculation of the Uncertainty of Calibrator Values for IVD Manufacturers," can be accessed on-line via www.cstl.nist.gov/ivdworkshop/index.html.

Copyright ©2003 IVD Technology