Table of contents
At a glance
How do you validate Azure DevOps as a source of GxP test evidence?
Validate how the Azure DevOps services, permissions, integrations and retention rules support the intended evidence process. Link requirements and test versions to results, the test environment and the delivered build. Include negative tests for missing tests, unauthorised changes and deletion. A successful pipeline run does not replace assessment of the results or an authorised decision on use.
- Services and servers have different operational and functional boundaries.
- Test cases, test points, and specific execution results must remain distinguishable.
- The technical validity of a report depends on the selection, version, and reference unit.
- Work items, runs, artefacts, and audit events require a coordinated retention strategy.
1. Assess the validation process instead of the product name
Azure DevOps can manage tasks, capture source code, build software, and aggregate test results. For GxP evaluation, it is essential to determine which of these data elements in your operation support a quality-related decision. An informal development board serves a different function than a test management system whose report is used as a key validation justification. Even a supporting tool can become relevant through such usage.
The present case pertains to computerized systems used in the manufacture of human medicinal products within the EU GMP framework. Annex 11 is the applicable GMP guideline for this system context. Section 4.7 specifically addresses the documented evaluation of the suitability of automated testing tools and testing environments. This does not lead to a blanket product approval nor to a uniform audit scope for all services.
Describe initially: Which statement does the Azure DevOps team make, who is responsible for that decision, and what failure mode could compromise this decision? Specific points of concern include a misfiltered report, an incorrectly overridden expected result, or a result generated for the wrong build. Additionally, document which evidence is generated outside the platform.
2. Services, Servers, and Licenses to be specifically defined
Azure DevOps Services is the continuously updated service operated by Microsoft. Azure DevOps Server runs on your own infrastructure and maintains its own release and patch schedule. Compare the applicable Microsoft release notes for your server with the actual installed version; a product designation alone does not indicate patch status or supported features. Server 2022 remains separately listed in the documentation.
Capture organization or collection, projects, process model, extensions, identity binding, and deployed build agents. For servers, include operating system, database, installation, and patch management. For services, establish a controlled approach to service changes and ongoing responsibility for the components. Identical screen labels do not imply identical functionality.
For test plan activities, standard view access and any project membership is insufficient to cover all required tasks. Microsoft documentation specifies that certain administrative and configuration tasks require either Basic + Test Plans or equivalent Visual Studio subscriptions, along with additional permissions. Prior to test planning, verify the actual access rights of testers and delegates. Having a released account does not confirm the user's substantive authority to make decisions.
3. Define requirements through to release
Microsoft describes linkages between requirements, tests, defects, and code changes. Value is realized only when these connections address your specific traceability question. A general relationship between two work items may be insufficient if it remains unclear whether the test covers the current requirement version or only an earlier draft.
The following work matrix is a proposed guidance document for a small development department. It outlines the required interdependencies and does not specify any particular additional product. Review the chain in reverse: from the actually installed package to the relevant testing activities and their underlying acceptance criteria.
| Object | Specific reference | Assessment question |
|---|---|---|
| Requirement R-12 | Locked material must not be selected; defined state | Is the accepted procedure clearly defined and valid throughout the inspection period? |
| Test case T-21 | Steps, test data, and expected rejection for R-12 | Can you identify the expected result that applied at the time? |
| Test point | T-21, Suite S-5, Browser configuration and tester | Is the intended implementation clearly assigned? |
| Execution and Results | Implementation against Build B105 in Test Environment Q1 | Do the actual results, time, data, and equipment match the expected outcome? |
| Build and Package | Source, dependencies, and identifiable delivery package | Is exactly this package the subject of the validation assessment? |
| Delivery of the software | Target environment, configuration, and documented decision | Is the installed package identical to the assessed package? |
4. Separate test cases and execution results
A test plan references test cases. If a commonly used test case is modified, this change affects all other plans and suites that reference it. Therefore, a reused plan name does not constitute a frozen test specification. Define how the currently used version of steps, parameters, and expected results remains traceable for future review.
A test point combines a test case, suite, configuration, and tester. Its repeated executions generate different results. The current Execute view displays the most recent result. For evaluation of a specific batch release, you may need to refer to the execution history rather than solely the most recent view.
Review a change in the own workflow after completion of execution: Can it be established whether the actual requirement was indeed verified? Is a subsequent run clearly distinguishable from the previous result? Prior to execution, determine when copies, a controlled export, or other predefined references are required. The appropriate procedure must also take into account shared steps and parameters.
5. Link pipeline success to decision-making on usage
A pipeline can successfully conclude despite including only a subset of the required tests. Therefore, define which tests run automatically within the pipeline, which are conducted outside the pipeline, and how reliable decision-making emerges from both domains. An unpublicized or incorrectly attributed result dataset must not be implicitly considered passed.
For Azure DevOps Services, Microsoft documents resource checks outside of the YAML file. These are managed by resource administrators. Self-approval is configurable; certain checks may be bypassed with appropriate administrative authority. The mere presence of an approval step therefore neither addresses the question of authorization nor establishes the technical or meaning in the process of the approval.
Define a unique decision for your distribution: Which package is authorized for which intended use in which environment? Take into account open deviations and operational prerequisites. A technical deployment authorization may be part of this process. Whether it simultaneously constitutes sufficient electronic signature depends on the specific intended use and must be evaluated separately.
6. Review changes and administrative exemptions
Repos can enforce branch policies. However, Microsoft explicitly distinguishes between bypassing rules when closing a pull request and when directly pushing. Review these permissions in conjunction with the rights to modify rules or pipeline resources themselves. A carefully configured validation loses its purpose if the same uncontrolled pathway can permanently disable it.
The test scenarios should reflect the actual permitted and prohibited actions associated with each role: modify test instructions, retrospectively record results, trigger release, adjust release rules, and perform an emergency override. A small organization does not require a fictitious standard number of signatures. It requires clear authorizations, appropriate controls, and reasonable responsibilities.
Azure DevOps Auditing is currently in Public Preview, disabled by default and tied to a Microsoft Entra organization; the server is not shut down. Evaluate the specific event log. Work Item History, Repository History, Organization Audit, and Entra Logs address various questions. A single export is not automatically the full consolidation of these records.
7. Results and context regarding the required deadline obtained
Microsoft states that when a pipeline run is deleted, associated data such as logs, build artifacts, test results, and run metadata are also lost. Test and release retention is not automatically synchronized; package types such as NuGet or npm are governed by separate retention rules. Organizational audits retain events by default for 90 days prior. A blanket setting of "everything is preserved" may fail to address the distinct requirements across these different domains.
Prepare a data inventory specifying owner, storage location, retention purpose, and required context. This includes, for example, test protocols, result records, failed trials, deviation decisions, package identity, and relevant associated changes. Set deadlines based on the supported activities and current documentation requirements, not on a predefined product value.
Review a typical old export or restoration record: Are the relationships still clear, are the attachments readable, and are versions identifiable? A PDF report can be useful for overview purposes. It only replaces further required data if its content and relevance for the intended use are adequately preserved.
8. Example: 100% passed, but two evidence gaps
The following example was not executed within an Azure DevOps organization. A small GMP application is intended to correctly handle material status, permissions, and export functionality. For Build B105, three test cases are planned across two browser configurations each. Each combination employs the same test suite and exactly one designated tester: a total of six test points.
Five existing results are classified as "passed". However, one of these results was generated only against B104. Transferability to B105 has not yet been evaluated. The sixth test point was not performed. Therefore, the example decision only applies to results that are directly applicable to B105.
- Make the missing run and old build reference visible; retain original results.
- For the B104 validation assessment, evaluate the impact of changes and justify the decision for acceptance or re-evaluation.
- Perform the missing Chrome export test or provide a technically justified amendment to the test protocol prior to making a decision.
- Combine the final assessment report with the actually released package and remaining deviations.
| Test case and required outcome | Configuration | Current Status | Assessment for B105 |
|---|---|---|---|
| T-21: Reject restricted material | Edge | B105, passed | Passed for the relevant scope |
| T-21: Reject restricted material | Chrome | B105, passed | Passed for the relevant scope |
| T-22: Reject unauthorized changes | Edge | B105, passed | Passed for the relevant scope |
| T-22: Reject unauthorized changes | Chrome | B104, passed | Transferability open; not counted here |
| T-23: Export including correct status | Edge | B105, passed | Passed for the relevant scope |
| T-23: Export including correct status | Chrome | Not performed | Evidence is missing |
9. Manage workload through a limited, operable scope
A SME should first fully simulate a representative validation process. For this, a subject-matter expert, a system owner, testers, and the required development and administration expertise are needed. QA participates in accordance with its defined responsibilities. Critical are available decisions and clear justifications, not an excessive number of formally named roles.
Effort drivers include primarily individual process templates, customizations, multiple projects, external testers, internal agents, distributed result sources, and previously unclear retention policies. Manufacturer documentation can substantiate functionalities and limitations. Assignment to specific use cases, configuration, and internal end-to-end testing remain to be finalized.
In operation, changes to test procedures, report queries, agent images, extensions, and deployment rights must be evaluated. For servers, planned patches are additional considerations; for services, service modifications and relevant announcements must be incorporated into the respective change management process. Targeted re-assessment of affected traceability chains is required. A consistently green status alone does not confirm that a modified report continues to convey the same conclusion.
10. Checklist prior to use as validation justification
The following work aid consolidates the decision points. It is not a Microsoft certificate and does not constitute a general mandatory catalog. Each response should refer to a specific project status. Open items will be documented with responsible party and impact on the intended use.
- Intended use and GMP-related claims clearly described.
- Services or servers, installed extensions, agents, and relevant license limits are recorded.
- Statuses of requirements, test cases, and configurations are assigned to the respective run.
- Selection and reference size of reports including missing tests understood.
- Build, delivery package, and target environment aligned throughout the decision process.
- Practical verification of change, deletion, bypass, and administrative rights.
- Technical pipeline status differentiated for professional usage decision.
- Storage for test data, runs, batches, and relevant histories aligned.
- Export or restoration with readable audit context validated.
- Define how changes are assessed during operation, who is responsible and who provides cover.
Project experience and collaboration
Bind release evidence to the build actually delivered
With the test team, development and QA, we trace a requirement to the test version, execution result and intended build. We prepare controlled counterexamples to see whether an old passing test or missing execution is visible in the overall status. Clear links and access controls may suffice for a limited evidence process. Extensive integrations require additional reconciliation. We support traceable mapping and retention. Neither a green pipeline status nor our technical check replaces the decision on operational use.
Frequently asked questions about Azure DevOps
Must Azure DevOps be validated as a complete suite?
The scope aligns with the intended GMP usage. If test plans and pipelines generate decision-relevant validation evidence, their relevant functions, interfaces, and controls must be adequately justified. This does not automatically imply that every unused feature requires the same level of testing. Additionally, describe which errors are detected through independent controls and which assertions are fully dependent on the tool itself.
Can an existing test plan remain unchanged as a baseline?
Not solely based on its name. A test plan references test cases; changes to shared test cases also impact other plans. Define how the actually used version of steps, parameters, and expected results is maintained. Verify, based on a completed execution, that a subsequent reader can clearly distinguish the original test instruction from the current test case.
What distinguishes test case, test point, and test run?
The test case describes the evaluation. A test point links it to suite, configuration, and tester. A run comprises specific execution results. Repeated executions of the same test point generate additional results; therefore, the most recent display does not necessarily constitute validation evidence for a prior release. This distinction prevents a current "passed" status from inadvertently replacing a previously non-compliant state.
Sources: [3]
Does a green pipeline suffice for release?
We confirm only that the configured version has met its technical success criteria. Additionally, assess technical coverage, correct build reference, open deviations, and operational readiness. An approval may support the prescribed workflow. The decision to be made and the authority responsible for it must be clearly defined in the respective procedure; automatic GMP release does not follow.
Must all requirements be automatically linked to test results?
You require a clear and robust assignment structure for your traceability purposes. Microsoft provides linkages and reporting capabilities whose scope depends in part on selected requirements and pipeline stages. Targetedly assess requirements without associated testing, as well as tests at incorrect stages. A manual catch-up step may be part of the process, but must be fully controlled and must not introduce undetected gaps.
Does the organizational audit cover all relevant changes?
No, not fully. Azure DevOps Auditing covers a specific set of events; Entra login and group modifications are recorded in other logging areas. For your use case, additional requirements may include work item, repository, and test result data. Review representative changes along with user details, timestamp, and context. The documented service function is currently in Public Preview and is not available for Azure DevOps Server.
Which access rights are particularly important?
Consider both technical changes and the alteration or circumvention of controls. For Repos, bypass rights for PR closure and direct push are separately relevant. This also includes administrative and resource rights. Test with actual roles to ensure that unauthorized actions are prevented or appropriately controlled according to the intended procedure. A role name alone is not sufficient evidence of effective restrictions.
Will test records be retained if a build run is deleted?
Not at all. Microsoft mentions test results and further evidence information among the data removed when deleting a run. Test, release and package storage must be considered separately. Therefore, before using it, you determine as decisive proof which data must be retained over which time period and how a later examiner finds the complete required context.
Can the same validation be applied to services and servers?
Requirements and test scripts may be reused if appropriate. However, the documented functionality and operational coverage must align with the current product state. For example, the checks documentation employed specifically applies to services, while servers have their own release and patch pathway. Evaluate differences in hosting, identity, enhancements, and operational responsibility before reusing existing results.
What should a small QA and IT team start with?
Select a typical usage decision and trace its validation evidence comprehensively from the approved requirements through to the installed package. Intentionally include a missing test, an outdated build reference, and an unauthorized change attempt. This results in a specific gap list. Only thereafter can a meaningful determination be made regarding which configuration, documentation, and additional testing are required for the specific scope.
Sources and review date
Source date: 6 October 2026. Focus: Azure DevOps Services; server instances and functions explicitly limited to services are treated separately. Public documentation does not replace an internal assessment of your organization's configuration or licensing. Regulatory reference: current EU GMP guidelines applicable to the described human medicinal product use.
- Microsoft: What is Azure DevOps?
- Microsoft: Create and manage test plans
- Microsoft: Test objects and terms
- Microsoft: Requirements traceability
- Microsoft: Define approvals and checks
- Microsoft: Access, export, and filter audit logs
- Microsoft: Set retention policies for builds, releases, and tests
- Microsoft: Branch policies and settings
- Microsoft: Azure DevOps Server release notes
- European Commission: EudraLex Volume 4
- European Commission: EU GMP Annex 11 – Computerised Systems
- European Commission: EU GMP Chapter 4 – Documentation
Sources checked: .
Verify Azure DevOps documentation purposefully
We support you in defining the intended tool usage, the traceability chain, and specific test cases for your organization.
- Cloud and SaaS validation →
Maintain the validated state of cloud and SaaS systems as suppliers issue new versions. We define the boundary between your responsibilities and the supplier’s and agree how each release will be assessed.

