AI Bias Testing Under the EU AI Act: What Regulators Expect

Regulators are not waiting for harm to occur before asking questions about AI bias testing under the EU AI Act. If you deploy a high-risk AI system, the obligation to detect and address discriminatory outputs is already baked into Articles 9, 26, and Annex III of the regulation. Fail to show evidence of bias monitoring, and you are not just non-compliant — you are exposed to fines of up to €15 million or 3% of global annual turnover under Article 99.

This post breaks down exactly what regulators expect, which protected characteristics matter, and how a small or mid-sized company can run credible bias testing without a team of data scientists.

# Why Annex III Systems Trigger Bias Obligations

The EU AI Act sorts AI systems into risk tiers. High-risk systems are listed in Annex III, and they include systems used in recruitment and CV screening, credit scoring, access to education, benefits decisions, and law enforcement risk assessment.

If you are an HR director whose company uses an AI tool to rank job applicants, your system almost certainly sits inside Annex III. Same goes for a financial services COO whose platform uses algorithmic credit scoring, or an education provider relying on AI-driven adaptive assessments.

Being a deployer (not the vendor who built the system) still carries direct legal weight. Article 26 is explicit: deployers must monitor system performance, report serious incidents, and ensure the system is used within the conditions defined by the provider. Bias is a performance issue. Regulators treat a system that systematically disadvantages people based on gender, ethnicity, or disability as a malfunctioning system, full stop.

# What "Bias Testing" Actually Means in Practice

Bias testing is not a single checkbox event. Regulators — including the Dutch Autoriteit Persoonsgegevens (AP), which has already signalled its intention to act as a lead supervisory authority for AI in the Netherlands — expect a continuous monitoring cycle with documented evidence at three stages.

# 1. Pre-Deployment Testing

Before a high-risk AI system goes live, you need baseline bias measurements. This means asking your vendor for disaggregated performance data: how does the system perform across gender subgroups, ethnic backgrounds, age brackets, and disability status? Article 9 of the regulation requires that risk management systems identify and analyse known risks, including bias risks, before deployment.

If the vendor cannot give you disaggregated metrics, that is itself a red flag. Article 27 gives deployers the right to request a summary of technical documentation from providers of high-risk systems. Use it. Document the request and the response.

Pre-deployment testing should produce at least:

• A baseline fairness metric (e.g. demographic parity difference, equalised odds, or predictive parity) for each protected characteristic relevant to your use case
• A written assessment of whether those metrics meet your acceptable threshold
• Sign-off from a senior decision-maker, not just IT

# 2. Ongoing Monitoring After Go-Live

The regulation does not let you test once and forget. Article 26(5) requires deployers to monitor system operation "on the basis of instructions of use" and to take action when outputs deviate from expected performance. Bias drift — where a system becomes more discriminatory over time as input data changes — is a real phenomenon, documented across credit, hiring, and healthcare AI systems.

A practical monitoring schedule for an SME:

• Monthly: run your fairness metrics on recent decision outputs using the same protected class breakdowns as pre-deployment
• Quarterly: review whether your protected class definitions remain appropriate (EU anti-discrimination law covers gender, racial or ethnic origin, religion, disability, age, and sexual orientation)
• Annually: conduct a full bias audit, which at minimum involves rerunning pre-deployment tests on current system outputs with fresh data

Keep the records. An Autoriteit Persoonsgegevens inspector will ask to see them.

# 3. Protected Class Coverage

EU anti-discrimination law, as codified in Directive 2000/43/EC and Directive 2006/54/EC, protects: racial or ethnic origin, sex, religion or belief, disability, age, and sexual orientation. Your bias testing must cover the characteristics relevant to your deployment context.

For HR tools, gender and ethnicity are the highest-priority characteristics. For credit scoring, age and disability status often surface as key risk areas. Do not test only on the characteristics that are easiest to measure. Regulators will notice the gaps.

# Tools You Can Use Without a Data Science Team

Two open-source libraries are widely accepted in European regulatory discussions as credible starting points.

IBM AIF360 (AI Fairness 360) provides over 70 fairness metrics and 11 bias mitigation algorithms. It supports tabular data, which covers most HR and credit scoring use cases. The documentation is solid and the community is active.

Microsoft Fairlearn integrates directly with scikit-learn and Azure Machine Learning, making it practical for teams already using Python. Its dashboard gives a visual breakdown of model performance across subgroups, which is useful when presenting results to non-technical stakeholders like HR directors or compliance officers.

Both tools produce output that can be exported and stored as part of your technical documentation under Article 11 of the AI Act. That documentation must be kept for at least 10 years after the system is taken out of service.

One important caveat: these tools measure statistical fairness. They do not capture qualitative harms, context-specific discrimination, or the lived experience of the people affected. Software output is evidence, not the whole picture.

# When to Bring in an External Auditor

Not every company needs to hire an external auditor immediately. But there are clear triggers that make external audit the right call.

Trigger 1: Your system makes decisions that significantly affect employment or credit. For recruitment AI, credit scoring, and benefits eligibility systems, the stakes are high enough that internal testing is rarely sufficient on its own. An external auditor brings independence that a regulator will take more seriously.

Trigger 2: You have found a bias problem and fixed it. Documenting a remediation with an independent verification strengthens your position significantly if a complaint is ever filed.

Trigger 3: Your vendor is pushing back on documentation requests. If a provider refuses to supply disaggregated performance data or technical summaries, you have a conflict of obligations. An external auditor can help you establish what the system is actually doing, and that documentation protects you regardless of what the provider claims.

Trigger 4: You are preparing for a Fundamental Rights Impact Assessment (FRIA). Under Article 27, deployers of certain Annex III systems must carry out a FRIA before deployment. A bias audit is not the same as a FRIA, but the two processes are deeply connected. External auditors can help run them together efficiently.

External bias audits in the Netherlands typically run between €5,000 and €25,000 depending on system complexity. That range sits well below the minimum fine for a serious violation under Article 99.

# The AVG/GDPR Interaction You Cannot Ignore

Bias testing often requires processing data about protected characteristics. In most EU jurisdictions, racial or ethnic origin and health data are special category data under Article 9 of the GDPR (AVG in the Netherlands). You cannot simply run disaggregated bias tests without a lawful basis for processing that sensitive data.

The practical solution is to work with anonymised or pseudonymised datasets for testing, or to rely on explicit consent or a legitimate public interest basis documented in your records of processing activities (RoPA). Your Data Protection Officer should be part of the bias testing design process, not a reviewer after the fact.

The AP has made clear in its published guidance that AI systems with discriminatory outputs will attract joint scrutiny under both the AVG and the incoming AI Act supervisory framework. These are not two separate compliance exercises. They are one.

# Your Next Step

Bias testing is one of five core compliance obligations Khairos AI Comply helps European SMEs structure, document, and evidence. Run the free 2-minute compliance check at comply.khairos.ai to see which obligations apply to your specific AI systems and where your documentation gaps are likely to be.