Medical Device Link.

Originally Published IVD Technology June 2003

IVD DIRECTIVE

A performance evaluation with good results can send an IVD to market; poor performance during an evaluation may send it back to square one.

Francesco Dati

Francesco Dati, PhD, is president of IVD Consulting (Marburg, Germany). He can be contacted at f.dati@t-online.de.

Developing evidence that an IVD achieves the performance stated by its manufacturer is an integral part of meeting the requirements of the European Union's Directive on In Vitro Diagnostic Medical Devices.¹ Often, complex studies are required to provide data demonstrating that an IVD achieves the stated performance and will continue to perform properly after it has been shipped, stored, and put to use at its final destination.

The IVD Directive classifies as IVDs all reagents and reagent products, calibrator materials, and instruments (including specimen receptacles) intended for use in the in vitro examination of human tissue, blood, or fluid samples for the purpose of providing information about a patient's state of health. The directive further divides IVDs into five regulatory categories, based on the intended use of the products and the relative risks to public health or patient treatment in the event they fail to perform as intended. For each of these categories, the directive establishes the essential requirements that the IVDs must satisfy.

Before such products may be CE marked, marketed, or put to use in the European Union (EU), the manufacturer must evaluate their performance in order to prove that they satisfy the essential requirements of the directive. In general, this means that the manufacturer must demonstrate that the IVD under consideration does not compromise the health or safety of patients and users, and that it has been designed and manufactured suitably for its stated medical purpose, taking into account the generally acknowledged state of the art.

In order for the CE mark to be affixed to an IVD product, the directive requires that the manufacturer develop evidence that the IVD under consideration achieves its stated performance. Furthermore, such evidence should also demonstrate that the IVD is designed, manufactured, and packed in such a way that its characteristics and performance during its intended use are not adversely affected under storage and transport conditions, taking account of the instructions and other information provided by the manufacturer (see sidebar, "Demonstrating Conformity").

Because of the importance of ensuring that diagnostic tests are performed correctly, the IVD Directive emphasizes the need for adequate labeling information and instructions for use, both for devices intended for professional use and for self-testing products used by laypersons.

Definition of Performance Evaluation

A performance evaluation is an investigation concerning the performance of an IVD, which may be based upon preexisting data, scientific literature, or new performance studies carried out for the purpose of setting or verifying performance claims. A performance study is one that involves the systematic collection of experimental data under the anticipated conditions of an IVD's use. Such studies may be carried out in the manufacturer's laboratories or in appropriate independent laboratories selected by the manufacturer.

In the IVD Directive, Annex III specifies 13 areas that manufacturers must address in their product's technical documentation.¹ Among these is a requirement for adequate performance evaluation data to support the performance claims of the manufacturer.

Consequently, before an IVD manufacturer may affix the CE mark to its product and place the product on the EU market, it must first carry out studies designed specifically to support its performance claims for the intended use of the IVD. Such studies are also required when the manufacturer of an IVD already authorized to carry the CE mark begins to market the product for a new intended use.

Performance studies are typically intended to assess an IVD's analytical and diagnostic sensitivity, analytical and diagnostic specificity, accuracy, repeatability, reproducibility, detection limits, and assured traceability. Such studies should also be designed to assess the potential effects of interfering factors (e.g., influence of a serum or plasma sample matrix, storage conditions, specimen treatment, or heat inactivation) and substances (e.g., anticoagulants, hemolysis, or hyperlipemia). Following are basic definitions for several types of commonly used performance studies.²

• Establishment Studies. This term predates the implementation of design control requirements in European and U.S. medical device regulations. Manufacturers generally carried out establishment studies during the product development phase in order to characterize the performance of the IVD under development.
• Validation Studies. Manufacturers generally carry out validation studies in order to determine whether the design of the IVD under consideration satisfies user requirements as expressed in the product's design input. Such studies examine all sources of variability relevant to the user.
• Verification Studies. Such studies are generally carried out to develop evidence that the IVD under consideration has met a particular performance specification or set of specifications.
• Demonstration Studies. Generally carried out by a testing laboratory, demonstration studies show the capability of an IVD test system to attain expected performance levels.
• Software Validation. As a general rule, software used in data management rather than diagnosis is not considered to be an IVD. However, software may be an accessory to an IVD test system, and as such may need to be registered as an IVD in its own right. Software validation is a critical tool used to ensure the quality of IVD software and software-automated operations.

Devices for Performance Evaluation

The IVD Directive divides IVDs into five regulatory categories, including a distinct classification for devices that are undergoing performance evaluation. This classification includes any device intended by the manufacturer to be subject to one or more performance evaluation studies in clinical laboratories or other appropriate environments outside its own premises. Regulatory procedures for handling such devices are set out in Annex VIII of the directive.¹

Although devices for performance evaluation are not required to carry the CE mark, the manufacturer must notify the appropriate national competent authority of its intent to conduct performance evaluation studies for a product. Such submissions must include a statement drawn up by the manufacturer or its authorized representative, confirming that all the essential requirements of the IVD Directive have been met excepting those that are under investigation as part of the evaluation. The statement should include the following information.

• Data allowing identification of the IVD under consideration.
• Evaluation plan, including the particular purpose and scope of the performance evaluation, and the number of IVDs concerned.
• List of laboratories taking part in the study.
• Starting date and scheduled duration of evaluation.
• For self-testing devices, the location and number of lay participants.
• Statement that the device conforms to the IVD Directive and that every precaution for health protection has been taken.

To accomplish the last of those, manufacturers should prepare a checklist of the IVD Directive's essential requirements and their product's corresponding technical documentation. Compiling such information during the earliest possible stages of product development will help to ensure that the manufacturer does not experience delays while additional conformity testing is performed.

In some EU countries (Greece, Iceland, Ireland, Italy, Lichtenstein, Luxembourg, The Netherlands, Portugal, Switzerland, and the UK) transitional arrangements apply, and performance-evaluation IVDs are allowed to be used without following the requirements of the directive.

In other EU countries (Austria, Belgium, Denmark, Finland, France, Germany, Spain, and Sweden) Annex VIII procedures must be followed for all performance evaluations conducted to CE mark a product, and the product must be labeled as a device for performance evaluation.

Manufacturers sometimes provide prototype or alpha-series instruments to external organizations that carry out safety or performance testing. Such prototypes may undergo a number of changes prior to large-scale production. These changes can be regarded as variants and referred to in the product's technical documentation. However, in the case of a substantial variation—especially if the risk to patient health and safety may be increased by the proposed changes—a risk analysis is needed to show substantial equivalence between the alpha instruments and the final series.

In circumstances where the manufacturer is uncertain whether sufficient performance evaluation data exist to demonstrate compliance with the essential requirements, so that the CE mark can be affixed to the product, discussion with the national competent authority (or with the relevant notified body, where applicable) may prove helpful before embarking on the planning of a large performance evaluation study.

Guidance for Studies

A manufacturer that is developing a new IVD typically compiles preliminary performance specifications using design input that incorporates user requirements. When the new IVD is introduced to the market, the manufacturer issues a final set of performance claims (sometimes also including claims of clinical significance), which may differ from previous internal specifications.

If an evaluation is necessary to demonstrate that the product achieves the performance claimed, the manufacturer should take into account various guidelines and scientific recommendations for conducting such evaluations.^3–7 Although such recommendations provide useful strategies, they cannot take into account the full diversity of IVD products or their broad fields of use. The manufacturer must adapt such recommended performance study protocols to the nature and intended use of the IVD under consideration. For instance, the performance evaluation of a high-risk IVD used especially for screening blood products (that is, any of those listed by the directive in Annex II, List A) will necessarily be far more expensive and time-consuming than an evaluation for any other type of IVD.

Where a performance evaluation is required, the UK Medical Devices Agency (London) recommends that the study should satisfy all of the following goals.

• Performed on the basis of an appropriate study protocol with well-defined goals.
• Provide sufficient data to support the performance claims of the IVD made in the information for use supplied by the manufacturer.
• Include sufficient healthy persons and subjects with different diseases and reflect the aims of the investigation.
• Examine appropriate features involving performance, diagnostic use, and safety.
• Include the compilation of a final written report containing a critical evaluation of all the data collected during the study, with appropriate conclusions.⁸

Manufacturers that are carrying out performance evaluation studies to validate product claims according to the requirements of the IVD Directive should pay particular attention to the recommendations of the European standard titled Performance Evaluation of In Vitro Diagnostic Medical Devices (EN 13612), which was compiled by the European Committee for Standardization (see sidebar, "Guidance for Performance Studies").⁹ This standard offers guidance to help manufacturers design and conduct performance studies that will accomplish the following goals.

• To develop evidence of an IVD's safety and effectiveness that can be provided to notified bodies and national competent authorities.
• To compile adequate performance evaluation data originating from new studies or available literature, as required by the IVD Directive.
• To ensure that appropriate data are available to support the manufacturer in product liability litigation.

The following sections summarize the responsibilities and requirements for performance evaluations as specified in EN 13612.

Responsibility for Study Direction. Because of the specific nature and use of IVDs, as distinct from drugs or from medical devices in general, EN 13612 emphasizes the responsibility of the manufacturer for the overall quality and performance of its product. This responsibility extends also to the initiation and proper and scientifically based conduct of the performance evaluation. The manufacturer should therefore define the responsibility, authority, and interrelation of all personnel who manage, conduct, and verify the performance evaluation of IVDs, including the appointment of a study coordinator.

Preconditions. Before starting a performance study, various preconditions should be fulfilled. For instance, the manufacturer should ensure that it has established specifications for its IVD, that it has controlled production processes and conditions, and that it has established quality control release specifications for the product. Similarly, the manufacturer should ensure that it can provide sufficient IVD samples for the performance study, experienced and trained investigators, and other necessary resources.

Study Protocol. The performance study protocol defines the purpose, structure, and organization of the study. The protocol should specify especially the study objectives, evaluation sites, timetable, necessary information on product design, performance claims to be determined or verified, appropriate specimens, test procedures (including comparative assays), and type of records with all results and relevant data. For performance studies of instruments or systems, additional specifications should address maintenance and cleaning, infection risk, carryover effects, and software.

Documentation. The documentation of the performance study should be prepared and kept according to the principles for quality systems and the requirements of the IVD Directive.

Final Review. The study monitor should conduct a final assessment and review of the performance study. The review should consider whether the IVD met or failed to meet the manufacturer's performance claims, and may also provide recommendations for appropriate remedial measures.

Reevaluation. The manufacturer may initiate a reevaluation of a particular IVD. Such a reevaluation is typical when a marketed product has undergone changes of manufacturing process or design and such changes may affect the product's specifications, performance claims, or safety.

Test Specimens. When the specimens used in the performance evaluation of an IVD are residual materials from blood donations or from samples taken for diagnostic purposes, the manufacturer is usually not required to take additional measures for the protection and safety of test subjects.

Safety of Test Subjects. In cases in which specimens obtained by invasive collection procedures are intended only for use in a performance study, all applicable legal requirements should be observed. The investigator is responsible for informing test subjects about the purpose of the study and the potential risk of the sampling procedure, and for obtaining their written consent. Results obtained from a patient's specimen using the product under evaluation should never be used as a basis for undertaking medical action.

Common Technical Specifications (CTS)

To assist manufacturers in designing and manufacturing products in conformity with the essential requirements of the IVD Directive—and to ensure the consistency of the review process across the EU's member states—Article 5 of the directive called for the development of a set of common technical specifications (CTS). The CTS provide guidance on criteria for performance evaluation and batch release, and on the identification and use of reference methods and materials (including their availability for specific types of IVDs). Finally published in 2002, the CTS cover the high-risk IVD products identified by the directive in Annex II, List A, as follows (see sidebar, "CTS for the Performance Evaluation of Serological Assays").¹⁰

• Performance evaluation of IVDs for the detection, confirmation, and quantification in human specimens of markers of HIV infection; HTLV-I and -II; and hepatitis B, C, and D.
• Performance evaluation of IVDs for blood-group determination.
• Manufacturer's batch-release testing of serological assays and blood grouping devices.

For this group of products, compliance with the CTS is examined by the selected notified body as part of the design dossier review, and is mandatory to satisfy the essential requirements of the directive so that a product can receive the CE mark. Proof of compliance with the CTS should be given in a product's technical documentation. Notified bodies always confirm the compliance of IVDs specified in Annex II, List A, as well as of IVDs that are subject to batch-release testing.

The NB-MED Recommendation Group, a consortium involving EU notified bodies and manufacturers, has issued recommendations on the application of the CTS for assessing the sensitivity and specificity of IVDs.¹¹ The recommendations include notes on the analysis of seroconversion panels as part of the performance evaluation of screening or confirmatory assays. The following paragraphs summarize key elements of the recommendations.

Sensitivity. According to these recommendations, during the performance evaluation of a new IVD the analytical and diagnostic sensitivity of the new product should be determined in direct relation to a comparison IVD that reflects the generally accepted detection level—the state of the art. A set of specimens that reflects the target population and contains the target marker should be tested using the new and the comparison IVD, ideally in parallel.

Discrepant test results between the two IVDs should be resolved either by evaluation of the respective sample in further test systems, by use of an alternative method or marker, by reviewing the patient's clinical status and diagnosis, or by testing of follow-up samples. On the basis of these data it should be possible to resolve most discrepant results.

Analytical sensitivity is usually determined by testing serial dilutions of reference materials. In cases where they are available, recognized national or international reference standards should be used.¹¹

Detection Limits. The consistency of a product's detection limits for the known genotypes and subtypes of a pathogen should be determined by serial dilution of samples containing known amounts of the genotypes and subtypes. The performance evaluation should reflect the prevalence of the pathogen in the target population, and assessment should be made in comparison with the levels of detection generally accepted at the time of testing.¹¹

Use of Seroconversion Panels. A seroconversion panel consists of a series of samples obtained from an individual during the early phases of infection. Such samples are usually obtained from plasma donations of subjects who have been freshly infected by the pathogen. The serial samples of a seroconversion panel reflect normal disease progression, and therefore indicate significant changes in the measured parameter, usually from negative to positive. As a result of differing regional practices or regulations (some of which are based on ethical considerations), the time intervals between the serial samples of a seroconversion panel may vary.¹¹ NB-MED is very direct in its recommendation that manufacturers of high-risk IVDs should carry out testing using seroconversion panels, stating as follows:

Detection of samples at very early stage of infection is an important feature and a major field for IVD improvement. During the early phases of infection, pathogen components are present and low-titer and low-avidity antibodies are initially generated. The presence of such antibodies constantly increases with time and may decrease again at later stages of the infection. For example, in the case of blood donation screening assays, the earliest possible detection of infection is very important to prevent transmission to blood donation recipients. To verify the suitability of an IVD for screening purposes, performance evaluation on seroconversion panels must be carried out.¹¹

Suitability for Blood Screening. After assessing the performance of the new IVD in several seroconversion panels, the overall performance of the new product should be compared with that of the comparison IVD. Taken together with an assessment of the test's diagnostic sensitivity, this comparison is the principal basis for determining the suitability of a new IVD for blood screening purposes.¹¹

Genotype Sensitivity. In order to ensure that a new IVD is capable of detecting all of its pathogen's genotypes and known subtypes, NB-MED recommends comparative testing using true-positive genotyped and subtyped samples that can provide information about potential assay-specific lack of sensitivity. Taking into account the prevalence of the pathogen in target populations, the comparative test results are the basis for determining whether the performance of the device achieves the levels of detection generally accepted at the time of testing.¹¹

Availability of Reference Materials

Annex III of the IVD Directive requires that performance evaluation data developed to support the performance claims of the manufacturer should themselves be supported by a reference measurement system when available.¹ The technical file should include information on the reference methods, the reference materials, the known reference values, and the accuracy and measurement units used.

It will be a major ongoing challenge for the IVD community to make available reference materials, seroconversion panels, and critical samples (e.g., isoforms of proteins or subtypes and rare variants of viruses) sufficient to permit standardized evaluation of antibody and antigen assays. The single members of reference seroconversion panels should be native specimens from single individuals stored frozen in aliquots and defined in the terms of geographical origin, type of donor or patient (including clinical status), nature of the specimen (serum, plasma, others), positivity criteria, and method of preservation.

Such reference panels with defined areas of application and performance criteria should be established by experts of the EU member states designated by national competent authorities and other interested parties. It would be extremely helpful if a specific organization would agree to be in charge of establishing a bank of such reference materials and panels. This activity would include the collection and preparation of suitable starting materials, as well as the storage and distribution of the panels, which should be made available to all interested parties.

Conclusion

In order for an IVD product to carry the CE mark, the IVD Directive requires that the manufacturer provide evidence in its technical documentation that the product performs as claimed, whether these claims are of technical, analytical, or diagnostic nature. Such evidence may be based on data already available to the manufacturer, on previous scientific literature, or on data originating from performance evaluation studies in a clinical or other appropriate environment in accordance with the product's intended use.

The CE mark indicates that the product meets the requirements of current legislation in each member state regarding the implementation of the IVD Directive. Furthermore, the CE marking of an IVD means that it has been produced under a quality system incorporating good manufacturing practices, and that the product fulfills its performance claims.

However, the CE mark alone will surely not be accepted as proof of the medical quality of a product. To demonstrate the medical benefits of an IVD product, laboratorians and other healthcare decision makers will expect to see evidence developed in performance studies and clinical trials. To document the cost-effectiveness of an IVD, they will expect to see data from health economics analyses. In these regards, technology assessment and evidence-based medicine are likely to play an ever-greater role in determining the access of IVDs to healthcare systems and their diffusion following introduction.

References

1. "Directive 98/79/EC of the European Parliament and of the Council of 27 October 1998 on In Vitro Diagnostic Medical Devices," Official Journal of the European Communities L 331 (1998): 1–37.

2. JS Krouwer, "Evaluation of Assay Systems: Cost-Effective Solutions for Today's Clinical Laboratories," Clinical Laboratory News 27, no. 3 (2001): 10–14.

3. R Haeckel and I Püntmann, "Validation of Measurement Procedures," In Vitro Veritas [on-line] 2001 [cited 7 May 2003]; available from Internet: http://www.acclc.es/invitroveritas/vol2/art20.html.

4. Preliminary Evaluation of Quantitative Clinical Laboratory Methods, NCCLS guideline EP10-A2 (Wayne, PA: NCCLS, 2002).

5. J Buettner, "Evaluation of Diagnostic and Prognostic Measures," in Evaluation Methods in Laboratory Medicine, ed. R Haeckel (Weinheim, Germany: VCH, 1993): 1–46.

6. R Haeckel et al., "Recommendations for the Multicentre Evaluation of Analytical Systems in Clinical Chemistry," in Evaluation Methods in Laboratory Medicine, ed. R Haeckel (Weinheim, Germany: VCH, 1993): 46–69.

7. J Ingrand and C Naudin, "Special Aspects in Immunoanalysis," in Evaluation Methods in Laboratory Medicine, ed. R Haeckel (Weinheim, Germany: VCH, 1993): 117–130.

8. EC Medical Devices Directives: Guidance Notes for Manufacturers on Clinical Investigations to Be Carried Out in the UK (London: Medical Devices Agency, 1996).

9. Performance Evaluation of In Vitro Diagnostic Medical Devices, EN 13612 (Brussels: European Committee for Standardization, 2002).

10. "Commission Decision on Common Technical Specifications for In Vitro Diagnostic Medical Devices," Official Journal of the European Communities L 131 (2002): 17–30.

11. Assessment of the Sensitivity of In Vitro Diagnostic Medical Devices: Guidance on the Application of the CTS, recommendation NB-MED/2.5.5/rec. 4 (Brussels: Coordination of Notified Bodies Medical Devices on Council Directives [NB-MED], 2000).

Illustration by Marco Aguilera

Francesco Dati, PhD, is president of IVD Consulting (Marburg, Germany). He can be contacted at f.dati@t-online.de.

Copyright ©2003 IVD Technology