Regulations and compliance · Practical guide

GxP test tools: assessing fitness for intended use

Challenge a test tool with faulty inputs or known discrepancies to see whether it detects them. A comparison can return a pass simply because the tool never examined a column that matters.

Table of contents

At a glance

Do CSV test tools themselves need to be assessed?

Yes. Demonstrate that the tool is fit for the particular test task. A faulty script or comparison tool can miss a system deviation and give false confidence. Assess its significance, ability to detect failure and existing controls. The scope need not match validation of the main system being tested.

Scope and Applicability

Annex 11 Section 4.7 requires a documented assessment of the suitability of automated testing tools and testing environments. The specific form of evidence shall be determined by the task and the risk of a false test result.

Sources: [1]

Also check whether the test can detect a failure

Separate the expected result as much as possible from the same logic being tested. If the test script inherits the same erroneous calculation unchanged, the two identical errors validate each other. An appropriate independent comparison baseline can mitigate this weakness.

Ensure traceability between test setup, inputs, and results. Modifications to the script, libraries, or test environment may alter the validity of the results. A result lacking a verifiable tooling baseline will be difficult to interpret in subsequent inquiries.

Sources: [1]

The intended tool validation statement describes

A test tool can perform actions, compare values, or generate evidence. A fault in the tool may lead to an undetected system error or an incorrect result. Therefore, it is essential to specify the basis of its output and the implications of a false green result. Not every auxiliary script carries the same level of impact.

Annex 11 Section 4.7 requires a documented assessment of the suitability of automated testing tools and test environments. This does not imply that every individual script necessitates an identical, comprehensive documentation package as a dedicated application. The extent of documentation shall be proportionate to the intended use, complexity, and risk associated with the test conclusion.

Distinguish commercial tools, in-house scripts, and configurations. A validated base product may still produce inappropriate results due to a misconfigured filter. A simple in-house script can effectively perform a well-defined task, provided its function and limitations have been appropriately assessed. The product name alone does not establish suitability.

Sources: [1]

A tool account with rules and boundaries

Capture version, inputs, comparison logic, excluded fields, and output. Identify who evaluates changes and who reviews the result. The document should explain the actual purpose, not merely collect installation data. A reviewer must understand what a passed test confirms and what it does not.

Document assumptions about data format, encoding and sorting. A comparison can misreport or misalign values if the order is reversed. The correct key rule is often more important than a large number of successfully processed lines. The requirements follow the concrete comparison purpose.

Own work help: guide a tool file with rules and limits
Tool aspectQuestion to investigateEvidence needed
Entrance Was the correct, complete dataset processed? Quantity, key, or file reference
Rule Which property is being compared? Clear logic and tolerance
Exclusion What is deliberately overlooked? Justified Limit
Result Are differences completely visible? Report with individual and total references
Version Which batch generated the statement? Reproducible tooling alignment
Change Who maintains and verifies the rule? Lifecycle responsible party

Sources: [1]

Example: Comparison script ignores the critical status field

A script compares migration files and reports no differences. However, its field catalog does not include the batch status. A incorrect batch release information remains undetected. The tool functions exactly as programmed, but fails to meet the required validation criteria.

Prepare a small, known reference dataset. It contains unchanged rows, missing data entries, duplicate keys, and a deliberately incorrect status. Define expected findings independently. Execute the script and compare the report against the expected findings. This test thus verifies not only execution but also detection capability.

Upon completion of the field is updated, the review is repeated. Evaluate previous statements made under inappropriate system status. A new correct version does not automatically render prior green reports liable. The affected scope is determined based on actual usage and data risks.

Own work aid: Error cases for the comparison script
Reference caseState introduced in the testExpected tool inspection result
Positive comparison Identical dataset No false difference
Missing line Object target missing Visible missing identification
Dual key Two target objectives have the same identifier. Non-unique identifier detected
Incorrect status Critical field out of specification Content assessment with record reference
Sorting Same data, reordered No unexplained misassignment
Reporting limit Many well-known differences No silent cut of important results

Sources: [1]

Independent target values and negative cases are used

A second copy of the same logic does not constitute a strong independent reference. Derive expected results from an appropriate scientific basis or from a separately validated method. For simple rules, a clear manual calculation may suffice. More complex tools require corresponding validation evidence.

Test failures of the tool itself. What happens if a file cannot be read, a column is missing or the format is invalid? An interrupted run must not appear as a successful report with zero findings. Users should be able to tell whether the check ran to completion.

Consider tolerances and normalizations. Intentional rounding may be appropriate for one purpose, but could mask critical differences for another. Document the rationale for why a rule is applicable. The default setting of a tool is not an automatic professional acceptance threshold.

Sources: [1]

Evidence and reproducibility obtained

Link tool version, input version, settings, and results report. A subsequent run with a different file or library version may generate different results. The exact technical implementation may vary; what is critical is the traceable retrievability of the actually used version.

When screenshots or other evidence are automatically generated, verify their correct referencing. A screenshot without a record identifier or featuring an outdated screen state provides little credible proof. A tool that performs actions swiftly must still enable the actual required intended-versus-actual comparison.

Assess manual interventions. If a user filters or corrects reports post-hoc, the final review conclusion must remain traceable. A modified result must not be issued as an unaltered automatic output. Roles and review responsibilities shall be defined in accordance with risk.

Sources: [1]

Proactively manage and reuse changes

A tool can be used across multiple projects if its purpose and boundaries are well defined. Review new data formats, fields, and rules. A proven comparison for simple master data does not automatically validate historical audit data or complex relationships. Reuse requires an appropriate context of application.

Changes will result in the affected reference cases being repeated and new risks being added. A minor code modification can impact the central comparison logic. Therefore, the number of modified lines is not a sufficient measure of scope. Identify the technical impact and appropriate regression testing.

Sources: [1]

Draft: intentionally prevent a wrong green comparison

Build a small reference dataset for the form-filling example: identical data, a missing data point, a duplicate key, and an incorrect status. Pre-define the expected findings independently of the script. Also verify how the tool handles a missing column or limited report output. Neither circumstance may go unnoticed and result in a successful overall status.

Sample filling example – internal work aid
tool fieldExample entryAcceptance question
Entrance Files with known volume. Was the correct dataset processed?
Key Unique record reference. Are duplicates detected?
Critical field Status is explicitly compared. No unjustify ed exclusion?
Failure scenario Known deviation or missing column. No false green statement?
Report Individual results and totals. Is the output complete and attributable?
Status Script, rules, and settings. Is the result retrievable and explainable?

Sources: [1]

Next steps

  • Define the inspection task and potential incorrect conclusions.
  • Evaluate the tool using known correct and incorrect cases.
  • Check the version and required operating instructions.
  • Evaluate changes to the conclusions of previous tests.
Further reading:

Project experience and collaboration

Test the comparison tool against known errors

We test your test or comparison tool against a small dataset with independently determined findings, such as a missing record, a duplicate key and an incorrect status. This reveals whether a green overall result actually covers the property that matters. A simple calculation aid may need a focused test. A reusable comparison script with many rules requires broader assurance. We record purpose, version and limits so your team can explain the result and retest after changes.

Frequently asked questions about Test tools

Does a small comparison script require the same validation scope as a specialist application?

No. For automated testing tools and test environments, Annex 11 Section 4.7 requires a documented assessment of suitability; the depth of this assessment depends on the specific use case. A limited script can be reasonably evaluated using appropriate reference data, failure scenarios, version references, and review. An automated decision filter requires a different evaluation approach than a simple schedule list. The impact of a false test result is particularly significant.

Sources: [1]

Is a known commercial test product available?

His reputation alone does not establish the suitability of your configuration and validation logic. Appropriate supplier evidence can support the evaluation.

Sources: [1]

Are manually written inspection checklists exempt?

No. Even in-house scripts can contain systematic errors. Their specific application and detection capability must be understood.

Sources: [1]

Does a successful run without error notifications suffice?

No. It first demonstrates execution. Verify with known positive and negative cases whether relevant errors are detected and fully reported. A tool may compare incorrect fields without technical error reporting. Therefore, professional error recognition capability is a central element of the tool's suitability assessment.

How do we test a report summarization or distribution limit?

Use a known inventory with sufficient expected differences and compare individual results as well as totals. Assess whether a limit is visible and whether the full required scope remains achievable. Otherwise, a successfully executed script may lose important results from the report.

May we carry over tolerances from a template?

Only after verification of their technical suitability. A tolerance may be appropriate for a measurement parameter and inappropriate for a characteristic or status. Document the purpose and rule. A tool's default value is not an automatic acceptance criterion for your specific validation evidence.

Sources and review date

The sources provide the technical foundation. Tables, checklists, and examples are derived methodological tools, not verbatim regulatory requirements.

  1. European Commission: EU GMP Annex 11, Computerised Systems (2011)Annex 11, January 2011: No. 4.7 regarding documented evaluation of automated testing tools and testing environments. Sample test cases and documentation selection in the article are independent implementation proposals.

Sources checked: .

Daniel Herrmann, Daniel Herrmann Consulting

Daniel Herrmann Consulting

From an open point to a traceable approach.

We support you in assessing the current situation, clarifying responsibilities, and implementing appropriate evidence within operations. The type of support that is most effective depends on your system, your team, and the specific objective.

  • Computer system validation →

    Bring your GxP systems into validated operation, from laboratory software, MES and eQMS to ERP. We define the scope, plan risk-based tests using GAMP 5 and hand the evidence to QA for release.

Send a project enquiry Customer testimonials →