Regulations and compliance · Practical guide

Draft Annex 22: how can you keep AI test data independent?

Renaming development data does not make them independent test data. Even new images of the same objects can distort the result. Track provenance, object identity, access and reuse. The 2025 draft Annex 22 provides a reference for preparation and does not impose a current GMP obligation.

Table of contents

At a glance

How do you recognise an independent AI test dataset?

Check whether the data and relevant items were kept separate from training and model selection, who had access in advance and whether test results influenced development decisions. Document provenance, partitioning and each use. Draft Annex 22 sets out specific expectations. The selected data should also represent intended use. Separation alone does not make an unsuitable dataset representative.

  • Separate development data, model selection, and final performance testing in a traceable manner.
  • Verify items and data provenance, not just file names or folder paths.
  • Evaluate repeated use to determine whether it influences development decisions.

1. What the draft addresses – and what currently applies

Source date is 7 October 2026. Annex 22 is available as a consultation draft; the 2025 consultation did not establish it as the current guideline. For specific human medicinal product GMP applications, the applicable existing requirements remain decisive. The following work aids support forward-looking planning.

The draft concerns static, deterministic models in critical GMP applications. Its provisions cannot simply be extended to continuously learning or generative systems. Even with an unchanged model, the conditions of use may change.

In machine learning, a validation dataset is often used for model selection. This is not the same as GMP validation of the entire system usage. Therefore, name the datasets according to their actual purpose, rather than simply labeling them as "Validation" in multiple folders.

Sources: [1] [2] [3] [4]

Further reading:

2. Determine which unit must be independent

In images, the relevant unit may be a physical object, while in time series, it may be a run or a continuous production segment. A random division of individual files may then distribute related observations across both sides. The model may be evaluated at an object whose characteristics it already knows.

The draft expressly considers physical objects and the independence of their characteristics. Alongside each file ID, record a suitable source or object ID. Justify the level at which you separate the data and identify any relationships that remain.

Describe also permitted similarities. The same professionally defined defect characteristic may occur during development and testing if it is intended for use. Crucially, it must be determined whether the testing provides new, independent observations that adequately reflect the intended use.

Sources: [1]

3. Maintain a record of origin and usage

A dedicated register links data collection, segregation, and access. It should demonstrate which data contributed to specific decisions. A directory established retrospectively cannot erase prior access, but it does make remaining uncertainties visible.

The following work aid can be integrated into existing data management systems. It does not require any specific tool. What is essential is that the information remains verifiable and is not based solely on memory.

Support document: How independence can be justified
entrySpecific statementAssessment question
Origin Object O-014, Location S-1, Acquisition Group G-07 Does the object or a closely related dataset exist during development?
Purpose Final test for model M-3 Was this acquisition already used to select M-3?
Distribution Fixed object list prior to development start Was the order silently rearranged later?
Access Person, Time Point, Purpose and Available Copies Could the developers access the test dataset?
Use Test run T-02, result report and follow-up decision Has the result triggered a model or threshold adjustment?

Sources: [1]

4. Example: Duplicate object identifications distort the test separation

Assuming 200 object identifiers are captured from two viewing angles each, a total of 400 images are generated. For the example, a identifier-based split is predefined: 140 identifiers with 280 images allocated for development and 60 identifiers with 120 images designated for the final test. It is initially assumed that each identifier refers to a distinct object. These figures are computational assumptions and do not constitute a statistically sufficient sample.

Data review reveals a flaw in this assumption: 10 of the 60 test identifiers refer to the same physical items as identifiers in the development inventory. These items were renumbered and photographed during a later data capture series. Consequently, 20 of the 120 test images are affected. After exclusion, 50 test items remain, each with 100 associated images. The calculation accounts for the overlap; however, a sufficient residual review has not yet been performed.

The team must evaluate the root cause, the actual usage during development, and the remaining validity of the data. It is not acceptable simply to remove the 20 images and claim the test is thereby automatically valid. Subgroups or rare failures could be inadvertently eliminated from the test as a result.

Own required matrix for object-based separation
VerificationExpected result in this exampleFind and Consequence
Split by identifier 140 development identifiers + 60 test identifiers = 200 identifiers Verify whether different identifiers actually refer to different items.
Number of images 280 + 120 = 400 images Quantity adjustment supplemented, but does not replace object verification
Object overlap No physically previously used item present in the test batch 10 affected objects identify; effect assess
Remaining balance 50 objects / 100 images per distance Representativeness and testing scope to be justified, not automatically accepted

Sources: [1] [4]

5. Separation requires appropriate access rules

For an early split, the draft addresses protection against prior access by development staff, including copies and access records. A locked master folder provides limited evidence if the same data were previously available as a download or presentation.

Define who prepares, labels and uses the test data. They need enough knowledge of the process to do the work, while preventing information from the test set from influencing development decisions. In small teams, assess actual overlap between roles. Different job titles do not establish independence. Section 6.5 of the draft proposes separating people with test data access from training and validation of the same model. Where that is organisationally impossible, they should only work on those activities alongside a colleague without such access. Put this two-person safeguard into practice.

In a separate control case, assess an unauthorized access and a permitted test access. Document the boundaries: not every prior information contact can be excluded by current authorization records.

Sources: [1]

6. Assess reuse based on its impact

Feedback can influence development each time a test dataset is used. If a failure identified in a test report is used for retraining, a further run on those data needs a different interpretation from a technical repeat with the model unchanged.

The draft requires a traceable record of usage. Methodologically, specify the decision derived from each run: restoration of a discontinued test, new pre-processing, alternative decision threshold, or new model. This enables targeted assessment of the impact on independence.

Do not draw a general conclusion that each data set may be used only once. Justify which statement remains valid for an additional run and where new independent data are required. This decision must precede the final evaluation of the model's performance.

Sources: [1]

Further reading:

7. Preparation prior to the final test

The functional area and QA define the scope of use and relevant errors; data specialists clarify origin and dependencies. IT supports access and traceable provisioning. The majority of effort typically arises from missing object references or undocumented prior usage, not from creating an additional folder.

  • Identify the intended operating conditions and important subgroups.
  • Set separation unit: file, object, run or other established unit.
  • Document the origin, labelling and permitted pre-processing.
  • Define responsibilities and roles prior to critical use.
  • Check object overlaps, copies, and prior accesses.
  • Associate each test run with the model version and subsequent decision.
  • Assess uncertainty and adequate coverage prior to final release.

Sources: [1] [4]

8. Avoid two misleading conclusions

"Independent" does not automatically mean "representative". A completely new inventory can omit rare defects or altered reception conditions. Therefore, check both the separation and the suitability for the intended statement.

Even a perfect result cannot compensate for a weak data foundation. If the evaluation is based on known items, ambiguous labels, or cases selected post hoc, it requires a different interpretation. Maintain such limitations open at the conclusion, rather than concealing them behind a high hit rate.

Sources: [1] [4]

Project experience and collaboration

Verify test data independence beyond file names

With process and data owners, we investigate whether the held-out test set was genuinely independent. Different filenames or identifiers can refer to the same item, and access during development matters. For new data, separation and ownership can be planned from the start. For existing data, we reconstruct use and possible overlap first. We organise any required separation of roles or two-person control with your team. If affected records are removed, we reassess which subgroups the remaining dataset still covers.

Frequently asked questions about Independent AI test data

Is Annex 22 already applicable?

No. Annex 22 remains a consultation draft from 2025 as of 18 October 2026; the closing of the consultation does not bring it into effect. The text may be used for preparatory purposes. During this use, draft provisions should be clearly identified and distinguished from existing applicable requirements.

Sources: [2] [3] [1]

Is a separate folder for test data sufficient?

It facilitates organisation but does not verify data origin or prior usage. Check object overlaps, copies, and access rights. A later-locked folder does not make previously known test data retrospectively unknown.

Sources: [1]

Why can two photos of the same object be problematic?

They may share characteristics already seen during model development. Depending on the task, separate the data at physical object level. The draft considers physical objects and allows assessment of the independence of their characteristics. A rule based solely on files is insufficient.

Sources: [1]

Are 60 test units from the example sufficient?

This cannot be inferred from the example. The numbers illustrate only how a physical overlap distorts data partitioning. Even a cleaned inventory must adequately cover the intended subgroups and relevant variants. The size and statistical validity of the data must be independently justified.

Sources: [1]

Can developers view test results?

Test results must be available for evaluation. The test data themselves should be distinguished from this: According to draft section 6.2, no personnel involved in development or training had access to these test data prior to the training phase. If a model or threshold adjustment is derived from a test result, a subsequent test with the same batch must be re-evaluated. However, a batch previously used for optimization is not required to be re-evaluated independently.

Sources: [1]

Must all test data be replaced after each repetition?

No. Section 6.3 of the draft requires documenting which data were tested, when, and how frequently; the section does not propose a universal one-time rule. A technical reiteration must be evaluated differently from an optimization based on existing test cases. If data are used for model development, their independence for a subsequent final validation test is no longer assured.

Sources: [1]

What should we do if we discover an overlap?

Identify affected data and objects, determine their actual usage and assess the impact. Then verify whether the remaining or additional data still cover the intended usage. Simply removing problematic lines does not constitute a complete justification.

Sources: [1] [4]

How can a small team ensure traceable independence?

The draft calls for effective separation of personnel. Anyone with access to test data should not take part in training or validating the same model. If organisational separation is not feasible, section 6.5 discusses working with a second person who has no access to those data. Describe the roles, access rights and specific pairing. Giving someone a different job title does not in itself achieve the proposed separation.

Sources: [1]

Sources and review date

Source date: 7 October 2026. Annex 22 is a consultation draft. No enforceable additional obligations are derived therefrom. Human medicinal product GMP; the numbers and registers are independent methods, not statistical release or completed product testing.

  1. European Commission: Draft Annex 22 – Artificial IntelligenceDraft consultation 2025, dated 07.10.2026: no current Annex 22 guidelines applicable. Sections 1, 3–7 and 10; static deterministic models in critical GMP applications.
  2. European Commission: Consultation on Chapter 4, Annex 11 and Annex 22Official consultation period from 07.07. to 07.10.2025; the review on 07.10.2026 is final. The conclusion of the consultation does not constitute implementation.
  3. European Commission: EudraLex Volume 4 – current document inventoryReviewed on 07.10.2026: Annex 11 Revision January 2011; dedicated veterinary area applying Regulations 2025/2091 and 2025/2154 from 16.07.2026.
  4. European Commission: EU GMP Annex 11 – Computerised SystemsRevision January 2011, applicable since 30.06.2011; §§ 1, 4, 6–7, 9–13, 16–17. Human Medicines GMP Guideline; no draft version.

Sources checked: .

Daniel Herrmann, Daniel Herrmann Consulting

Daniel Herrmann Consulting

Test data with verifiable origin and clear purpose

We support you in defining the boundaries between development and test batches, assessing overlaps, and preparing a traceable inspection plan.

  • AI system validation →

    Bring your AI system into GxP use with traceable evidence and keep it under control in operation. Once the system and intended GxP use are defined, we validate according to risk, addressing data, supplier, tests, release and change control.

Send a project enquiry Customer testimonials →