Table of contents
At a glance
How do we assess automatically generated test evidence, including error detection and analysis?
Test whether the automation detects relevant failures and links results correctly, as well as whether it runs. Define independent expected values, controlled failure cases and the meaning of passed, failed and not executed. Retain evidence linked to the software, data and test rules. The use decision must account for actual coverage and unresolved findings. A green status indicator alone cannot make that decision.
- Subject, testing equipment, and result evaluation to be considered separately.
- Controlled errors demonstrate whether the test system detects relevant deviations.
- Assess digital evidence based on statement, assignment, and availability.
Separate three items from the inspection.
The first question relates to the production or QMS function being used. The second question concerns the tool that validates this function. The third question pertains to the analysis that derives a conclusion from individual test results. If the same erroneous calculation pathway generates both actual and expected values, the comparison may still display as green despite incorrect results.
Therefore, explain the source of the expected value and how it is independent of the test article. A manually traceable reference calculation may be appropriate for a limited case. For more extensive data, other suitable reference methods may be available. The key factor is the justified credibility of the assessment, not the blanket assertion that every independent test must be performed by a second person.
Sources: [1]
Classify automation and tool usage
The FDA-CSA Guidance from February 2026 supports appropriate tools and existing digital evidence. It addresses production and QMS software in the medical device sector. Its recommendations are not binding and cannot be issued as universal GxP requirements without a scope review. Applicable requirements remain decisive.
Annex 11 Section 4.7 requires a documented suitability assessment for automated testing tools and test environments. The FDA-CSA Guidance classifies tools that automate testing for production or QMS software as supporting software. Such tools are subject to validation within the specified medical device context, although the effort may be risk-based and reduced. General business or infrastructure tools without a direct role in production or QMS operations should be distinguished from this requirement. Therefore, the specific use case is more relevant than the tool name.
Map the full chain of checks
The own working matrix considers more than the actual test script. A correct script can run against the wrong environment, read outdated data or get a wrong overall result through a faulty summary. Such transitions belong to the evaluation of the proof.
For small teams, a concise representation with clear identifiers is often sufficient. It should clearly indicate which test run was evaluated under which software version and conditions. Avoid relying solely on a variable file name such as "latest run" for assignment. The required level of detail should be aligned with the specific statement and associated risk.
| Level | Relevant question | Appropriate control point |
|---|---|---|
| Test data | Are reference values and boundary conditions known? | Dataset status and independent expected justification |
| Execution | Was the intended function actually tested? | Assign environment, configuration, and software version |
| Comparison | Does the regulation identify relevant deviations? | Intentionally enter incorrect values and missing results |
| Summary | Can reviewers still see failures and cases that were not run? | Compare individual results against overall status |
| Retention | Can the required statement be reasonably traced or justified at a later date? | Associated result data, inspection rules, and context available |
Sources: [1]
Example: five controlled test conditions
In this example, a manufacturer reviews a calculation that converts an entered length in millimeters to centimeters. The assumed requirement is: Result = Input / 10; missing inputs must not generate a valid result. The internal references are 120 mm = 12 cm and 35 mm = 3.5 cm.
To evaluate the testing system itself, three controlled error conditions are specified: a false result value, a missing result, and a case that was not executed. The table describes the expected responses of the automation. It does not claim any product-specific tool functions.
| Case | Controlled input or output | Independently established expectation | Required statement of the test system |
|---|---|---|---|
| A | 120 mm; issued 12 cm | 120 / 10 = 12 | Comparisons for this evaluation |
| B | 35 mm; released 3.5 cm | 35 / 10 = 3.5 | Comparisons for this evaluation |
| C | 120 mm; intentionally issued 120 cm | Incorrect: Factor 10 is missing | Indicate a failed technical comparison; evaluate expected error detection separately |
| D | 35 mm; result missing | Not sufficient basis for results | Make the missing result visible and do not interpret it as zero |
| E | The step was not performed. | No inspection conclusion available | Do not identify executed; no statement "contained" about the technical function |
Sources: [1]
Verify the overall status against individual findings
In the expected case, two comparison results are correct. The other three conditions test error detection or the display of missing evidence. “5 of 5 passed” is meaningful if it explicitly refers to successful testing of the five control conditions. It would be wrong to interpret it as five correct business calculations.
Therefore, define the significance of the counters. Two successful technical comparisons and three correctly identified control conditions represent distinct parameters. Additionally, document how aborts, time overruns, and missing data are handled. A skipped test must not be considered passed solely due to the absence of an error message.
Sources: [1]
Retain the critical digital evidence
The FDA recommends leveraging appropriate existing digital records instead of unnecessarily duplicating them through manual documentation. Crucially, it must be determined whether the record provides sufficient objective evidence for the intended use. For an automated process, this may require establishing the linkage between test data, inspection rule, run, and individual result.
A screenshot of the green overview page rarely shows all the necessary information. Conversely, not every technical debugging log is automatically a required regulated record. Determine which information the proof decision bears and how it remains available. If the required records are kept electronically, their Part-11 applicability must be checked separately.
Sources: [1]
Assess modifications to the test system
The validation logic may also change. A new rounding rule, a different data format, or a modified comparison operator can impact the conclusions from previous runs. Address such changes explicitly and review the affected control conditions. In the example, an excessively large tolerance might inadvertently accept an intentionally incorrect value.
The appropriate level of control depends on the intended use. A one-off supporting script with independently verified results should be evaluated differently from a central test system whose status underpins numerous release decisions. A generic category based solely on the term "script" is equally inadequate, as is the assumption that technical tools are inherently irrelevant.
Sources: [1]
So assign CI/CD test evidence to the released build
You save time on recurring tests when the pipeline executes them and the original report provides traceable evidence of the required test performed on the same software package. If your release procedure accepts the report as valid documentation, you can directly evaluate the same test run and its results, eliminating the need for repetition and duplicate documentation.
When your pipeline tests are executed automatically, associate the test documentation with the specific software package. Generate the package once, perform testing, and use the same package during software release. A version name alone may refer to a different package upon a subsequent build.
For computerized systems in the EU GMP context, Annex 11, Section 4.7 requires a documented assessment of the suitability of automated testing tools and testing environments. Section 10 outlines the controlled implementation of changes in accordance with a defined procedure. The following matrix demonstrates how to consolidate evidence for a pipeline run.
| Reference | What you should retain | What you are aligning |
|---|---|---|
| Build and Run | Commit, Build-ID, Pipeline Run and Execution attempt, as well as the SHA-256 checksum of the software package. | The checksums for the tested package and the package submitted for release are identical. |
| Test environment and test data | Identification and configuration status of the environment, along with version and checksum of the test data. | The documented states correspond to the intended test configuration. |
| Test version and results | Test plan and test script version, expected critical cases, status of each case, and original report with checksum. | Report checksum and run order are consistent. Each expected mandatory occurrence is clearly documented and verified as completed and passed. |
| Release and Storage | Release decision with date and responsible person, optionally including deviations and reference to the evidence package. | Keep reports, assignments, and decision access restricted. Verify retrievability, readability, and integrity during the specified retention period. |
Perform a synthetic interface test to verify the release hold on the pipeline. The versioned test plan expects T01 for acceptance of a dataset containing batch identification and T02 for rejection of the same dataset lacking batch identification. First, execute both scenarios against the same package. Repeat the run under identical environment and identical test data. This time, exclude T02 during execution while keeping the test plan unchanged.
The report now shows only T01 as passed. The pipeline must detect deviations from the test plan, indicate T02 as missing or skipped, and block release. Verify the block also using a report without test cases and with a missing file. Retain the findings from negative runs as evidence of the block function.
In this procedure, a named, authorized individual makes the release decision independently of test execution. They verify the package assignment, required test results, and the evaluation of documented deviations. Record their decision in relation to the specific package and run.
Sources: [3]
Checklist prior to use of automated results
Our checklist is designed to support the decision of whether you can rely on a specific run. It is not a complete tool validation protocol. Start by considering the consequences of a false green result and select appropriate controls accordingly.
If a controlled error goes undetected, simply removing the affected case from the summary is insufficient. Investigate the root cause and the extent of impact. Previous results may be affected if the same erroneous test rule was previously applied. Document the resulting assessment and the corrected validation evidence.
- Test subject and tool used clearly demarcated.
- Measured values independently and technically justifiable determined.
- The assessment covers deliberately incorrect results, missing results and checks that did not run.
- The run is unequivocally linked to software, data, and validation rules.
- Individual results and overall status are consistent and non-contradictory.
- Required digital evidence available and assigned.
- Evaluated open findings and changes for the usage decision.
Sources: [1]
Select screenshots following the required statement assertion
A screenshot is appropriate when the visible user interface itself is part of the requirement—such as an observable warning, the correct signature intent, or an unauthorized action. For a comprehensive comparison across multiple data rows, a machine-readable record may provide more compelling evidence. Select the format that best reflects the observation required by an independent reviewer. A high number of screenshots does not constitute independent evidence of test quality.
Annex 11 requires appropriate justification for the intended use; this does not imply a general obligation to photograph every mouse click. Conversely, a log file is not a blanket substitute for any visible or interactive requirement. Assess the scope of the log: does it capture only a function call or also the actual observed system state? The FDA-CSA Guidance has a specific applicability domain for production and quality management system software in medical devices; it does not constitute a general exemption from required validation evidence for all pharmaceutical GMP tests. For automated testing tools and test environments, documented suitability assessment as specified in Annex 11, Section 4.7, remains required.
Sources: [3]
Own detection matrix: Observation prior to file format
In the example of a batch release test, an authorized person must be able to verify a specific record; an unauthorized user must not be able to do so. The technical log shows a rejected request. The user interface (UI) test further verifies that the interface clearly displays the correct status and does not falsely indicate success. Both records may be required together, as they provide complementary information. A standalone server message or a standalone screenshot would leave the other element unverified.
Identify the test object, status, test case, input, expectation, actual result, and timestamp in an appropriate manner. Not every piece of information needs to be present in each screenshot, provided the package ensures unique traceability. Maintain required digital evidence under controlled conditions. When a run is repeated, previous findings and new results must remain distinguishable. A subsequently generated image must not inadvertently be used as evidence of a prior test execution.
| What the test establishes | Relevant evidence | Boundary |
|---|---|---|
| Alert is understandable to the user | Controlled UI screenshot with test reference | Fails to prove a sole backend prevention measure |
| Unauthorized access is denied | Call and actual authorization status check | A hidden button is not sufficient. |
| All the specified values are correct | Traceable machine-readable comparison | Compared rules and population must be appropriate. |
| Signature pertains to the correct version | Record, Version, and Signature Reference | A decorative signature image is not sufficient |
| Error leads to correct overall status | Individual findings and summary | Green symbol alone is insufficient |
Sources: [3]
Project experience and collaboration
Trace the conclusion of the test run to the individual evidence records
Working with the test team and process owners, we trace a critical test conclusion through the input data, script version, execution and result assessment. We introduce controlled errors to see whether the overall status accounts for missing or failed steps. A simple calculation may need different evidence from a workflow involving both a user interface and an integration. We supplement the evidence accordingly. Suitability of the tool and environment remains a separate question. A successful test run alone cannot establish it.
Frequent Questions on Automated Test Certifications
Is a green run sufficient as validation evidence?
It may provide part of the evidence. Its significance depends on the inspection objective, coverage, data, and correct evaluation of results. A green status does not confirm unverified requirements and does not replace the decision on open findings or the intended use.
Sources: [1]
Must every test tool be fully validated?
Not every tool requires the same level of testing depth. In the EU GMP context, Annex 11, Section 4.7 mandates a documented assessment of the suitability of automated test tools and test environments. In the medical device sector, the FDA classifies tools that automate testing of production or QMS software as supporting software; such software must be validated, although the effort may be proportionate to its generally lower risk profile. Key considerations include intended use, potential impact of failures, and the tool's ability to provide traceable evidence.
Sources: [1]
Why should we intentionally input incorrect results?
This enables targeted evaluation to determine whether the test setup detects a relevant error. Without such control conditions, a comparison may remain green, as it fails to perform an effective validation. The selection of errors should align with key requirements and potential misclassifications.
Sources: [1]
Can the test and production function use the same formula?
This could compromise the independence of the reference comparison. If both comparisons incorporate the same erroneous logic, the deviation might remain undetected. Therefore, justify the use of an appropriate reference, such as an independently verified invoice for selected cases or another credible professional basis.
Sources: [1]
How do we handle a test that was not run?
He provides no indication of the intended behavior. Specify it accordingly and assess the impact on the decision on conformity. The absence of a defect does not constitute compliance if the inspection has not been carried out.
Sources: [1]
When do automated tests require additional screenshots?
When a relevant observation is not adequately supported by the other evidence, a log can fully capture calculations and status without requiring the display to be visible; conversely, a screenshot does not constitute a complete processing history. Select the appropriate documentation format based on the specific test statement and applicable requirements. A screenshot for each test step is not a necessary objective.
Sources: [1]
Are automatically generated logs always considered Part 11 records?
No. The key issue is whether the record constitutes required electronic records in the specific context. A log that supports the necessary validation evidence must be treated differently from a routine technical log lacking such a purpose. The use must be documented and justified.
Sources: [1]
What does 5 of 5 mean in the example of this article?
The five lines evaluate different control conditions. Two lines contain correct comparative values; three additional lines require visible error or missing states. A comprehensive conclusion must preserve this meaning. It must not appear as though five individual calculations have been successfully verified.
Sources: [1]
Must changes to a test script be evaluated?
Yes, to the extent they may influence the conclusions drawn from the used evidence. Review relevant rules, data formats, and summary results. Not every text change has the same impact; a modified tolerance or error handling may be significant.
Sources: [1]
Does this method apply to every GxP system equally?
The work aid is transferable, but the cited FDA-CSA scope is not automatically applicable. For pharmaceutical GMP systems or clinical applications, the relevant requirements must be specifically determined. The contribution neither asserts universal tool mandates nor claims a general exemption from validation evidence.
Can we retrospectively provide a missing image record?
A subsequent follow-up is a new observation and must be clearly identifiable as such. It must not appear as an original execution record. Assess the gap in evidence and establish a controlled supplementary or repeat testing procedure. The original finding, new execution, and conclusion must remain distinguishable.
Sources and review date
Source date: 15 October 2026. The FDA-CSA Guidance applies to manufacturing and QMS software used by medical device manufacturers; Annex 11 incorporates the EU-GMP domain. Criteria, numerical examples, and tables are provided as standalone methodological aids. These do not constitute a general requirement for screenshot documentation nor represent the actual review of a specific product.
- FDA: Computer Software Assurance for Production and Quality Management System SoftwareFinal guidance of 3 February 2026. Non-binding recommendations for production and quality management system software in the medical device sector.
- 21 CFR Part 820 – Quality Management System RegulationQMSR, effective from 2 February 2026: § 820.10, incorporating ISO 13485:2016 under § 820.7, and § 820.35 relating to records. The CSA Guidance directs for software validation to ISO subsections 4.1.6, 7.5.6, and 7.6, and for records to 4.2.5. These ISO references are substantiated by the FDA Guidance; an independent full-text comparison of the ISO standards is not claimed.
- European Commission: EU GMP Annex 11, Computerised Systems (2011)Annex 11, January 2011: No. 4.7 regarding appropriate test scenarios and documented evaluation of suitability for automated testing tools and test environments. No. 10 on controlled changes carried out according to a defined procedure. Specific evidence matrices in the article serve as standalone reference guides.
Sources checked: .
Jointly develop the specific evidence process
We help your team assess automated test systems and their digital records professionally, and systematically close remaining gaps.
- Computer system validation →
Bring your GxP systems into validated operation, from laboratory software, MES and eQMS to ERP. We define the scope, plan risk-based tests using GAMP 5 and hand the evidence to QA for release.

