Regulations and compliance · Practical guide

Draft Annex 22: which performance metrics do you need to assess an AI model?

A model can make many correct decisions while missing the errors with the most serious consequences. Translate those consequences into performance metrics and criteria set before testing. The worked example draws on the 2025 draft Annex 22. Its numerical threshold is an example chosen for this exercise.

Table of contents

At a glance

Which metric demonstrates that an AI model is fit for its intended use?

No single metric establishes suitability. Choose metrics that reflect the consequences of incorrect decisions, intended use and relevant subgroups. Set acceptance criteria before the decisive test and calculate metrics from traceable individual decisions. Draft Annex 22 lists possible metrics. It does not prescribe one universal minimum accuracy for every model.

  • First, define which error has which consequence for the process.
  • Evaluate overall performance and relevant subgroups separately.
  • Define acceptance criteria prior to testing and consider the uncertainty of the sample.

1. Establish the draft's status and the boundaries of intended use

On 7 October 2026, Annex 22 will continue to be treated as a draft consultation. The consultation ended in 2025, which did not result in an automatic entry into force. The following proposals are intended to prepare a concrete use of human medicine GMP in addition to the existing applicable requirements.

The draft concerns static, deterministic models in critical GMP applications. Turn its list of possible metrics into a test plan for your own use. First define the decision expected of the model and how a person or another process step contributes to it.

This article elaborates on performance assessment. The independence of the test dataset is addressed separately, as a properly calculated metric can still be misleading if based on known or inappropriate data.

Sources: [1] [2] [3] [4]

Further reading:

2. Define the failure before choosing a metric

Take a specific decision: An image is classified as "defect" or "no defect". A missed defect and a falsely rejected good part have different implications. The department must explain which error is relevant to product quality, further processing, and available inspection capacity.

Define, in addition, what "positive" means in your invoice. In the following example, "positive" means "defect detected". Without this definition, sensitivity and specificity can be correctly calculated but incorrectly interpreted in a professional context.

Define which reference determines the actual state. Uncertain or disputed reference evaluations constitute a separate quality issue. They should not be adjusted solely to align with the model outcome, merely to improve the key performance indicator appearance.

Sources: [1]

3. Select additional metrics instead of a total rate

The draft lists measures including sensitivity, specificity, accuracy, precision and F1. Choose measures suited to your decision and define their numerators and denominators. Explain which type of error each metric reveals.

The following table uses standard definitions for binary classification. TP denotes detected defects, FN denotes missed defects, FP denotes false positives (defective parts incorrectly flagged), and TN denotes correctly accepted good parts. The suitability of these definitions for other applications remains to be assessed on a case-by-case basis.

Work aid: Link metric and statement to accountability
MetricCalculationProcess question
Sensitivity TP ÷ (TP + FN) What proportion of actual defects is detected?
Specificity TN ÷ (TN + FP) What proportion of the good batches is correctly not flagged?
Precision TP ÷ (TP + FP) What proportion of reported issues are actual defects?
Accuracy (TP + TN) ÷ Total number What is the proportion of all correct decisions?
F1 2 × TP ÷ (2 × TP + FP + FN) How are precision and sensitivity linked in this summary?

Sources: [1]

4. Sample Calculation: 240 test cases fully evaluated

For the example, 240 independent test cases are assumed: 40 actual defects and 200 non-defective cases. Prior to testing, only two criteria apply to this example: sensitivity of at least 95 percent and specificity of at least 97 percent. The technical appropriateness and statistical validity of this assumption are not established.

The model detects 38 defects and overlooks 2; it also flags 8 good parts; 192 good parts are correctly identified. Thus, 230 out of 240 decisions are correct, amounting to approximately 95.83 percent. Despite this, one of the two assumed criteria is missed.

The matrix preserves the absolute values. Reporting only the overall percentage masks the differential impact of errors. Furthermore, the fulfillment of an observed percentage value would still not constitute sufficient evidence for a safe population-based conclusion.

Example: Results matrix with pre-defined target values
SizeCalculation and valueAssessment in this example
Defects TP 38 + FN 2 = 40 2 evaluate each overlooked defect individually
Conforming units TN 192 + FP 8 = 200 8 defective batches
Sensitivity 38 ÷ 40 = 95 % 95% confidence limit achieved
Specificity 192 ÷ 200 = 96 % Accepted limit 97% failed
Precision / F1 38 ÷ 46 ≈ 82.61 %; 76 ÷ 86 ≈ 88.37 % Supplementary description, no own acceptance criteria
Overall decision 230 ÷ 240 ≈ 95.83% Accuracy No acceptance based on overall percentage; specificity criterion not met

Sources: [1] [4]

5. Make relevant subgroups visible

A strong overall performance can mask a weak subgroup. In the example calculation, 30 defects could belong to a common defect type and 10 to a rare defect type. If all 30 common defects are detected, but only 8 of the rare defects, the overall sensitivity remains at 38 out of 40, or 95 percent; the rare defect group achieves 80 percent.

The draft describes predefined subgroups and suitable criteria. You need not test every possible combination to the same depth. Justify which differences matter to the intended use, such as material, site, device or failure type.

Avoid retrospective exclusion of only well-performing groups. A weakness identified during testing may serve as a basis for a new hypothesis and further investigation. It should not disappear due to a revised group definition in the final report.

Sources: [1]

6. Separate decision threshold from acceptance criterion

The decision threshold controls at what model value a case is flagged for investigation. The acceptance limit defines the measured performance level that should be met for the intended use. These are distinct specifications. Optimizing the decision threshold based on review of the final test results affects the independence of the performance evaluation.

Also assess the handling of unresolved or technically defective cases. Such cases must not be silently excluded from consideration. Priorly define how they are classified, investigated, and taken into account in the overall decision.

When a boundary is exceeded, a justified decision follows: investigate root causes, restrict the scope of application, evaluate additional controls, or revise the model. Simply adjusting the boundary post hoc to match the result does not provide credible evidence of suitability.

Sources: [1] [4]

Further reading:

7. Preparation for a credible performance qualification

Subject Matter Experts justify the consequences of errors and acceptable performance. Data specialists define calculation methods and uncertainty, while QA verifies the traceability of the decision. The roles require identical definitions; a purely technical performance export does not substitute for this alignment.

  • Define Intended Use, Positive Class, and Reference Assessment.
  • Identify relevant failure consequences and subgroups.
  • Define numerator, denominator, and handling of undecided cases.
  • Justify criteria prior to the critical test from a technical standpoint.
  • Evaluate the independence, representativeness, and scope of the test data.
  • Keep absolute case numbers, individual findings and computational path.
  • Consider failed criteria and remaining uncertainty in the decision.

Sources: [1] [4]

8. Write a decision instead of a list of metrics

The conclusion should specify for which intended use any particular statement is made. Identify the version that was reviewed, the key criteria, significant errors, and limitations. A model may be appropriate for a limited scope of application without necessarily demonstrating its suitability for the entire originally intended use.

Link the assessment to subsequent operations. Which metric is monitored further, which subgroup requires attention, and what signal triggers a renewed evaluation? Thus, test results serve as the starting point for controlled use rather than a single success metric.

Sources: [1] [4]

Project experience and collaboration

Examine critical errors behind the overall performance rate

High overall accuracy does not establish that a model suits your process. With process owners, we distinguish missed defects, unnecessary rejections and relevant subgroups. Data owners explain how the test set was assembled and which cases it lacks. A narrowly defined use may need a different combination of metrics from multiple sites or product variants. We agree acceptance criteria before the test, then examine whether the results support the intended use or require a narrower scope.

Frequently asked questions about AI performance metrics and acceptance criteria

Does Annex 22 require a specific minimum accuracy?

The proposed draft does not establish a universal minimum threshold. It outlines the selection of appropriate metrics and pre-defined criteria tailored to the specific application. The numbers presented in this article's example are internal assumptions and must not be construed as official regulatory limits.

Sources: [1]

Why can 95.83% accuracy in the example of this article not be sufficient?

Because the previously accepted specificity criterion of 97 percent is not met: 192 out of 200 batches achieve only 96 percent. The overall high rate does not justify this finding. Furthermore, the appropriateness of the criteria themselves must be substantiated.

Sources: [1]

Is sensitivity always more important than precision?

This depends on the consequences of errors and additional controls. Undetected defects and further deviations may give rise to different risks. Prioritization should emerge from the specific process, not from a general ranking of key performance indicators.

Sources: [1] [4]

What does it mean to have a small subgroup with poor performance?

It may point to a specific usage limit, but with a small sample size it provides a potentially unreliable estimate. Assess error type, data volume, and impact jointly. A good average should not conceal the weakness.

Sources: [1]

Can we select a different threshold after the test?

A change is possible but must be evaluated as a development or change decision. The performance observed in the previously assessed test batch cannot automatically constitute an independent conclusion. The further validation evidence should be planned accordingly.

Sources: [1]

How do we handle cases without a model outcome?

Determine in advance how technical failures or undecided cases are recorded and evaluated. A silent exclusion can embellish the performance. Whether manual processing is permissible and sufficient depends on the defined process.

Sources: [1]

Do 240 cases provide sufficient statistical reliability?

No. The volume merely illustrates the invoice. For a reliable conclusion, selection criteria, subgroups, defect frequencies, and required statistical confidence must be applied. An observed percentage does not guarantee future performance.

Sources: [1]

What should management learn from the results?

Which uses are supported by evidence, which criteria have been missed, and what limitations or additional controls are required. A single success rate is insufficient for this assessment. The decision should identify the associated effort and responsibility for outstanding items.

Sources: [1] [4]

Sources and review date

Source date: 7 October 2026. Annex 22 remains a consultation draft. All example quantities and limit values are assumptions, not regulatory requirements or claimed test results. No statistical sampling advice is provided for a specific product.

  1. European Commission: Draft Annex 22 – Artificial IntelligenceDraft consultation 2025, dated 07.10.2026: no current Annex 22 guidelines applicable. Sections 1, 3–7 and 10; static deterministic models in critical GMP applications.
  2. European Commission: Consultation on Chapter 4, Annex 11 and Annex 22Official consultation period from 07.07. to 07.10.2025; the review on 07.10.2026 is final. The conclusion of the consultation does not constitute implementation.
  3. European Commission: EudraLex Volume 4 – current document inventoryReviewed on 07.10.2026: Annex 11 Revision January 2011; dedicated veterinary area applying Regulations 2025/2091 and 2025/2154 from 16.07.2026.
  4. European Commission: EU GMP Annex 11 – Computerised SystemsRevision January 2011, applicable since 30.06.2011; §§ 1, 4, 6–7, 9–13, 16–17. Human Medicines GMP Guideline; no draft version.

Sources checked: .

Daniel Herrmann, Daniel Herrmann Consulting

Daniel Herrmann Consulting

Translate performance metrics into traceable decisions

We support you in consolidating root cause analyses, metrics, and acceptance criteria for your intended use and in clearly documenting the limitations of the evidence.

  • AI system validation →

    Bring your AI system into GxP use with traceable evidence and keep it under control in operation. Once the system and intended GxP use are defined, we validate according to risk, addressing data, supplier, tests, release and change control.

Send a project enquiry Customer testimonials →