AI Procurement: Assessing Third-Party AI Systems Before Deployment

When an organisation procures a third-party AI system, it does not transfer its governance obligations to the vendor. Under the EU AI Act, the organisation that puts an AI system into operation is the deployer — and deployers carry compliance obligations that exist regardless of who built the system. Most procurement processes are not designed for this reality.

The deployer problem in AI procurement

The EU AI Act defines a deployer as a natural or legal person that uses an AI system under its authority. This definition is technology-agnostic: it does not matter whether the system was built in-house, procured from a software vendor, embedded in a SaaS product, or delivered as an API. If you put it into operation, you are the deployer, and Article 26 applies to you.

This creates a procurement problem. A standard software procurement process evaluates functionality, price, security, data protection, and contractual terms. None of these standard evaluation criteria are sufficient to assess an AI system's compliance with the EU AI Act. The Declaration of Conformity, the technical documentation, the intended purpose, the training data disclosure — none of these appear in a typical RFP or vendor assessment questionnaire. The result is that organisations procure AI systems, deploy them, and only discover their governance gaps when a regulator asks, or when something goes wrong.

The vendor's conformity does not discharge the deployer's obligations. A vendor can truthfully state that their AI system has a valid EU Declaration of Conformity. This means the system met the requirements at the time it was assessed. It does not mean that your use of the system in your specific context is compliant. How you use the system — including whether you implement adequate human oversight, whether you use it for its stated intended purpose, and whether you maintain appropriate logs — is your responsibility, not the vendor's.

A structured AI procurement due diligence framework

The following framework organises procurement due diligence into five stages. It is designed for procurements of AI systems that may fall into the high-risk category under Annex III. For systems that are clearly minimal-risk, a lighter-touch version covering Stage 1 and Stage 2 only is proportionate.

Stage 1 — Pre-procurement: establish risk classification before evaluation begins

Before issuing an RFP or beginning vendor evaluation, the procuring organisation must determine the likely risk classification of the system it intends to procure. This is not the vendor's responsibility — it is the deployer's. A system that falls into the high-risk category under Annex III brings a set of compliance obligations that must be built into the procurement process from the start, not retrofitted after contract signature.

The Annex III high-risk categories most commonly encountered in standard organisational procurement are: AI systems used in employment decisions (recruitment, performance management, termination); AI systems used in education or training (admissions, assessment, monitoring); AI systems used in access to essential private services (credit scoring, insurance risk assessment); AI systems used in law enforcement or migration contexts (public sector procurers); and AI systems used in critical infrastructure management. Any procurement falling into these categories must follow the full five-stage framework.

Stage 2 — Vendor information request: what to require before evaluation

For high-risk AI systems, the following information must be requested from all vendors in scope before evaluation begins. Vendors who cannot provide this information in a high-risk AI procurement context are not in a position to meet their obligations under the EU AI Act — and deployers who proceed with such vendors take on the compliance gap themselves.

Required document	EU AI Act basis	What to check
EU Declaration of Conformity	Art. 47	Is it current? Does it reference the specific system version being procured? Is the conformity assessment body accredited?
CE marking confirmation	Art. 48, 49	Confirms the system has been through the conformity assessment process required for high-risk AI in the EU market.
Instructions for use (intended purpose statement)	Art. 13(3)	Is your intended use of the system within the stated intended purpose? Deployment outside the intended purpose constitutes misuse and shifts liability.
Technical documentation summary	Art. 26(6), Annex IV	Deployers are entitled to request access to relevant portions of technical documentation. At minimum: training data source description, performance metrics, and known limitations.
Human oversight capabilities description	Art. 14	What mechanisms does the system provide for human oversight? What can a human operator stop, override, or escalate? Is there an audit trail of human interventions?
Logging and record-keeping description	Art. 26(5)	What does the system log automatically? What can be exported? In what format? For how long are logs retained by the vendor?
Post-market monitoring report (if available)	Art. 72	Has the system been in operation elsewhere? What performance data is available? Have any serious incidents been reported?

Stage 3 — Contract negotiation: what must be in the contract

Standard software contracts are not adequate for high-risk AI procurement. The following provisions are required under the EU AI Act or necessary to protect the deployer's compliance position. Many vendors will resist some of these terms — the deployer must decide which are non-negotiable before entering negotiation.

Intended purpose warranty. The vendor warrants that the system performs as described in the instructions for use when used for the stated intended purpose. Deviations must be disclosed proactively.
Technical documentation access. The deployer has a contractual right to access relevant portions of technical documentation on request, including for regulatory purposes.
Notification of material updates. The vendor must notify the deployer of any update that materially affects the system's capabilities, risk profile, or intended purpose, with sufficient notice for the deployer to reassess its compliance position.
Serious incident notification. The vendor must notify the deployer within a defined period (recommend 72 hours) of becoming aware of any serious incident involving the system, including incidents involving other deployers' instances.
Audit rights. The deployer retains the right to audit the vendor's compliance with its obligations under the EU AI Act, or to commission an independent audit, with reasonable notice.
Data processing terms aligned to GDPR. If the system processes personal data (which most AI systems do), a Data Processing Agreement must be in place that meets GDPR Article 28 requirements. The AI Act and GDPR operate in parallel — compliance with one does not satisfy the other.
Decommissioning and data return. On termination of the contract, what happens to data processed by the system? What happens to logs generated during operation? The contract must address both.

Stage 4 — Pre-deployment review: what must happen before go-live

Before deploying a procured AI system into live operations, the following steps are required for high-risk AI.

Fundamental Rights Impact Assessment. Article 27 of the EU AI Act requires deployers in certain sectors — public bodies and private bodies deploying AI in education, employment, essential services, and several other categories — to conduct a FRIA before deployment. This is not the same as a Data Protection Impact Assessment, though the two may overlap. The FRIA specifically assesses the impact of the AI system on the fundamental rights of those affected by its decisions.

Human oversight mechanism design. The deployer must design and document how human oversight will work in practice: who reviews outputs, with what frequency, under what circumstances an output can be overridden, and what the escalation procedure is when an operator identifies a potential error. This design must be documented and communicated to all operators before deployment.

Operator training. Article 26(2) of the EU AI Act requires deployers to ensure that operators of high-risk AI systems have adequate AI literacy and receive appropriate training. Training must cover: what the system does and what it cannot do; its known limitations; how to identify and respond to anomalous outputs; and the escalation procedure.

Stage 5 — Ongoing monitoring: what must continue after deployment

Performance monitoring. The deployer must monitor the system's performance against its stated specifications throughout its operational life. This means defining what "performing as expected" looks like for this specific use case, establishing a threshold at which performance degradation triggers review, and having a process to act when that threshold is crossed.

Incident recording. Every incident — not only serious incidents — involving the AI system must be recorded in the organisation's incident management system and linked to the AI inventory entry for the system. Article 26(5) requires deployers to keep logs generated by the system; the deployer's own incident record is additional to, not a substitute for, the system's logs.

Annual vendor review. The ongoing relationship with the AI system vendor must be actively managed, not left to passive contract monitoring. The annual review should cover: has the system's risk classification changed? Has the vendor's conformity status been renewed? Have there been incidents or significant updates? Is the deployer still using the system for its stated intended purpose?

The most common procurement failure: gap between stated and actual use. The most significant compliance risk in third-party AI procurement is not vendor non-compliance — it is deployer scope creep. An organisation procures an AI system for recruitment screening, obtains the correct documentation, implements the required controls, and then — over time — extends its use to performance management, promotion decisions, and redundancy selection. Each extension moves further from the system's validated intended purpose and creates incremental compliance exposure that is not captured in any assessment. The intended purpose check must be part of the annual review, not a one-time procurement gate.