Bias and Discrimination in AI

Investigate AI bias through proxy variables and fairness metrics.

What Is Bias and Discrimination in AI?

Investigate potential bias in an AI resume screening tool using a fairness monitoring dashboard. Learn how proxy variables create indirect discrimination, understand fairness metrics like demographic parity and equalized odds, and practice the correct escalation response when discriminatory outcomes are confirmed in a high-risk AI system.

What You'll Learn in Bias and Discrimination in AI

Bias and Discrimination in AI — Training Steps

  1. Articles 10 and 15: Bias Testing and Monitoring

    The EU AI Act requires high-risk AI systems to be tested for bias - and the requirement goes deeper than checking for obvious discrimination. AI bias often hides in proxy variables: features that appear neutral but correlate with protected characteristics. Postal code can correlate with ethnicity or socioeconomic background. University name can correlate with socioeconomic status and access to opportunity. Employment gaps can disproportionately affect specific demographics. Biased AI outputs can constitute illegal discrimination even when the bias is unintentional. Articles 10 and 15 require providers and deployers to test for bias, monitor systems continuously, and take corrective action when discriminatory patterns emerge.

  2. Audit Alert

    An email arrives from the Internal Audit team. The quarterly AI audit has found statistically significant disparities in the TalentMatch screening tool's interview advancement rates.

  3. Fairness Dashboard Overview

    Alice opens the fairness monitoring dashboard via the link in the audit email. The overview shows interview advancement rates by demographic group and automated fairness metrics.

  4. Investigating Demographic Parity

    The Demographic Parity Ratio is 0.52 - well below the 0.80 threshold. This means Group C's advancement rate is barely half of Group A's. Alice needs to understand what is driving this disparity.

  5. The Proxy Variable Problem

    The investigation reveals the root cause. Two features dominate the screening decisions: 'university ranking' (weight: 0.34) and 'postal code' (weight: 0.28). Postal code shows a 0.72 correlation with Group C membership - meaning the AI is effectively using location as a proxy for demographic group membership.

  6. Equalized Odds Analysis

    Alice examines the second failing metric. The Equalized Odds Gap measures whether the AI treats equally qualified candidates equally regardless of group membership. A gap of 0.31 (threshold: 0.15) confirms that even among candidates with identical qualifications - same experience, same skills, same certifications - advancement rates differ significantly by group. The bias is systemic, not explained by qualification differences.

  7. Escalating the Issue

    The evidence is clear. The TalentMatch system produces discriminatory outcomes through proxy variables. Alice must formally flag the issue via the dashboard and escalate to the audit team with a clear recommendation.

  8. The Right Response to Confirmed Bias

    When bias is confirmed in a high-risk AI system, the correct response is to suspend the system pending remediation - not to adjust thresholds, add disclaimers, or schedule a review for next quarter. Under the EU AI Act, deployers must ensure high-risk AI does not produce discriminatory outcomes. Once bias has been identified and documented, continuing to operate the system creates knowing liability. Every application processed through a biased system after discovery is a potential discrimination claim. The path forward requires removing or de-weighting the proxy variables, retraining the model, and re-running the fairness evaluation before the system can resume operation.

  9. Key Takeaways

    Test for bias proactively Do not wait for complaints or regulatory audits to discover bias. Build fairness testing into your AI monitoring process from day one. Bias that goes undetected causes real harm to real people. Understand proxy variables Features that appear neutral - postal code, university name, employment gaps - can correlate with protected characteristics and produce discriminatory outcomes. Always evaluate whether your model's features could serve as demographic proxies. Bias testing is a legal requirement The EU AI Act makes bias testing mandatory for high-risk AI systems, not an optional best practice. Articles 10 and 15 require both providers and deployers to test, monitor, and mitigate bias. When bias is confirmed, suspend the system Continuing to operate a system after confirming discriminatory outcomes creates knowing liability. Suspend operations, remediate the root cause, and verify the fix before resuming. Document everything Maintain records of your fairness evaluations, identified issues, and remediation steps. This documentation is essential for demonstrating compliance to supervisory authorities.