Bias and Discrimination in AI

What Is Bias and Discrimination in AI?

Investigate potential bias in an AI resume screening tool using a fairness monitoring dashboard. Learn how proxy variables create indirect discrimination, understand fairness metrics like demographic parity and equalized odds, and practice the correct escalation response when discriminatory outcomes are confirmed in a high-risk AI system.

What You'll Learn in Bias and Discrimination in AI

• Identify proxy variables that create indirect discrimination in AI systems
• Interpret fairness metrics including demographic parity ratio and equalized odds gap
• Understand why bias testing is a legal requirement under the EU AI Act
• Practice the correct escalation response when bias is confirmed in a high-risk AI system
• Recognize the difference between adjusting thresholds and addressing root causes of bias

Bias and Discrimination in AI — Training Steps

1. Articles 10 and 15: Bias Testing and Monitoring

   The EU AI Act requires high-risk AI systems to be tested for bias - and the requirement goes deeper than checking for obvious discrimination. AI bias often hides in proxy variables: features that appear neutral but correlate with protected characteristics. Postal code can correlate with ethnicity or socioeconomic background. University name can correlate with socioeconomic status and access to opportunity. Employment gaps can disproportionately affect specific demographics. Biased AI outputs can constitute illegal discrimination even when the bias is unintentional. Providers must test high-risk systems for bias before release (Articles 10 and 15). Deployers must monitor them in use, and suspend them and inform the provider when they present a risk (Article 26).

2. Audit Alert

   An email arrives from the Internal Audit team. The quarterly AI audit has found statistically significant disparities in the TalentMatch screening tool's interview advancement rates.

3. Fairness Dashboard Overview

   Alice opens the fairness monitoring dashboard via the link in the audit email. The overview shows interview advancement rates by demographic group and automated fairness metrics.

4. Investigating Demographic Parity

   The Demographic Parity Ratio is 0.52 - well below the 0.80 threshold. This means Group C's advancement rate is barely half of Group A's. Alice needs to understand what is driving this disparity.

5. The Proxy Variable Problem

6. Equalized Odds Analysis

   The investigation reveals the root cause. Two features dominate the screening decisions: 'university ranking' (weight: 0.34) and 'postal code' (weight: 0.28). Postal code shows a 0.72 correlation with Group C membership - meaning the AI is effectively using location as a proxy for demographic group membership. Proxy variables are one of the most insidious forms of AI bias. The feature itself appears neutral - everyone has a postal code and attended a university. But when these features correlate with protected characteristics, they produce discriminatory outcomes through the back door. Alice examines the second failing metric. The Equalized Odds Gap measures whether the AI treats equally qualified candidates equally regardless of group membership. A gap of 0.31 (threshold: 0.15) confirms that even among candidates with identical qualifications - same experience, same skills, same certifications - advancement rates differ significantly by group. The bias is systemic, not explained by qualification differences.

7. Escalating the Issue

   The evidence is clear. The TalentMatch system produces discriminatory outcomes through proxy variables. Alice must formally flag the issue via the dashboard and escalate to the audit team with a clear recommendation.

8. The Right Response to Confirmed Bias

   When bias is confirmed in a high-risk AI system, the correct response is to suspend the system pending remediation - not to adjust thresholds, add disclaimers, or schedule a review for next quarter. Under the EU AI Act, deployers must monitor a high-risk system and suspend it once it presents a risk (Article 26(5)). Once bias has been identified and documented, continuing to operate the system creates knowing liability. Every application processed through a biased system after discovery is a potential discrimination claim. The path forward requires removing or de-weighting the proxy variables, retraining the model, and re-running the fairness evaluation before the system can resume operation.

9. Key Takeaways

   Test for bias proactively Do not wait for complaints or regulatory audits to discover bias. Build fairness testing into your AI monitoring process from day one. Bias that goes undetected causes real harm to real people. Understand proxy variables Features that appear neutral - postal code, university name, employment gaps - can correlate with protected characteristics and produce discriminatory outcomes. Always evaluate whether your model's features could serve as demographic proxies. Bias testing is a legal requirement The EU AI Act makes bias testing mandatory for high-risk AI systems, not an optional best practice. Providers must test for and mitigate bias before release (Articles 10 and 15), and deployers must monitor the system in use (Article 26). When bias is confirmed, suspend the system Continuing to operate a system after confirming discriminatory outcomes breaks the deployer's duty to suspend a system that presents a risk (Article 26(5)), and turns every later decision into a potential discrimination claim. Suspend operations, remediate the root cause, and verify the fix before resuming. Document everything Maintain records of your fairness evaluations, identified issues, and remediation steps. This documentation is essential for demonstrating compliance to supervisory authorities.