Bias and Discrimination in AI
Investigate AI bias through proxy variables and fairness metrics.
What Is Bias and Discrimination in AI?
Investigate potential bias in an AI resume screening tool using a fairness monitoring dashboard. Learn how proxy variables create indirect discrimination, understand fairness metrics like demographic parity and equalized odds, and practice the correct escalation response when discriminatory outcomes are confirmed in a high-risk AI system.
What You'll Learn in Bias and Discrimination in AI
- Identify proxy variables that create indirect discrimination in AI systems
- Interpret fairness metrics including demographic parity ratio and equalized odds gap
- Understand why bias testing is a legal requirement under the EU AI Act
- Practice the correct escalation response when bias is confirmed in a high-risk AI system
- Recognize the difference between adjusting thresholds and addressing root causes of bias
Bias and Discrimination in AI — Training Steps
-
Articles 9, 10 and 26: Bias Testing and Monitoring
High-risk AI systems must be tested for bias, and that goes beyond obvious discrimination. Bias often hides in proxy variables: neutral-looking features that correlate with protected characteristics. Postal code - ethnicity or socioeconomic background University name - socioeconomic status Employment gaps - specific demographics Biased outputs can be illegal discrimination even when unintentional. Providers examine data for bias (Article 10) and test before release (Article 9). Deployers monitor in use and, if they see a risk, suspend the system and inform the provider or distributor and the market surveillance authority (Article 26(5)).
-
Audit Alert
An email arrives from the Internal Audit team. The quarterly AI audit has found statistically significant disparities in the TalentMatch screening tool's interview advancement rates.
-
Fairness Dashboard Overview
Alice opens the fairness monitoring dashboard via the link in the audit email. The overview shows interview advancement rates by demographic group and automated fairness metrics.
-
Investigating Demographic Parity
The Demographic Parity Ratio is 0.52 - well below the 0.80 threshold. This means Group C's advancement rate is barely half of Group A's. Alice needs to understand what is driving this disparity.
-
The Proxy Variable Problem
-
Equalized Odds Analysis
The root cause: two features dominate screening, 'university ranking' (weight 0.34) and 'postal code' (weight 0.28). Postal code has a 0.72 correlation with Group C membership, so the AI uses location as a proxy for group. The feature looks neutral, but because it correlates with a protected characteristic it produces discriminatory outcomes through the back door. Alice checks the second failing metric. The Equalized Odds Gap is 0.31 (threshold 0.15): even candidates with identical qualifications advance at different rates by group. The bias is systemic.
-
Escalating the Issue
The evidence is clear. The TalentMatch system produces discriminatory outcomes through proxy variables. Alice must formally flag the issue via the dashboard and escalate to the audit team with a clear recommendation.
-
The Right Response to Confirmed Bias
When bias is confirmed in a high-risk AI system, the correct response is to suspend the system pending remediation - not to adjust thresholds, add disclaimers, or schedule a review for next quarter. Under the EU AI Act, deployers must monitor a high-risk system and, when they have reason to consider that its use may present a risk, suspend it and inform the provider or distributor and the market surveillance authority (Article 26(5)). Once bias has been identified and documented, continuing to operate the system breaches that duty and exposes the company to penalties. Every application processed through a biased system after discovery is a potential discrimination claim. The path forward requires removing or de-weighting the proxy variables, retraining the model, and re-running the fairness evaluation before the system can resume operation.
-
Key Takeaways
Test for bias proactively Do not wait for complaints or regulatory audits to discover bias. Build fairness testing into your AI monitoring process from day one. Bias that goes undetected causes real harm to real people. Understand proxy variables Features that appear neutral - postal code, university name, employment gaps - can correlate with protected characteristics and produce discriminatory outcomes. Always evaluate whether your model's features could serve as demographic proxies. Bias testing is a legal requirement The EU AI Act makes bias testing mandatory for high-risk AI systems, not an optional best practice. Providers must examine their data for bias (Article 10) and test before release (Article 9), and deployers must monitor the system in use (Article 26). When bias is confirmed, suspend the system Continuing to operate a system after confirming discriminatory outcomes breaks the deployer's duty to suspend a system whose use may present a risk and to inform the provider and the market surveillance authority (Article 26(5)), and turns every later decision into a potential discrimination claim. Suspend operations, remediate the root cause, and verify the fix before resuming. Document everything Maintain records of your fairness evaluations, identified issues, and remediation steps. This documentation is essential for demonstrating compliance to supervisory authorities. When this applies: the Digital Omnibus (Regulation (EU) 2026/1744) moved the deployer duties for Annex III high-risk systems such as hiring tools to 2 December 2027. Bias in hiring is unlawful discrimination today, so the right response is the same now.
Security Framework Coverage
NIST CSF
- PR.AT-01 Personnel are provided with awareness and training so that they possess the knowledge and skills to perform general tasks with cybersecurity risks in mind
NICE Framework
- OG-WRL-008 Privacy Compliance
- OG-WRL-002 Cybersecurity Policy and Planning
EU AI Act
- Art. 10 Data and data governance
- Art. 27 Fundamental rights impact assessment for high-risk AI systems
- Art. 26 Obligations of deployers of high-risk AI systems