AI Training Data Poisoning

What Is AI Training Data Poisoning?

Most enterprise AI does not answer from memory. It retrieves documents and repeats what they say, so whoever can write to the knowledge base can write the answer. You'll play both sides: edit a vendor policy and a test procedure from a compromised contributor account, then audit the assistant that cites them, following a confident answer back through document metadata, a version diff and the audit log. You'll separate pre-training poisoning from retrieval poisoning and apply the controls that catch each.

What You'll Learn in AI Training Data Poisoning

• Define data poisoning and distinguish between pre-training poisoning (corrupted training datasets) and RAG poisoning (manipulated retrieval databases)
• Identify behavioral indicators of a poisoned AI system, including contradictory guidance, unfamiliar source citations, and shifted recommendations
• Trace the causal chain from a poisoned document in the knowledge base to an incorrect AI-generated business decision
• Apply content integrity controls including contributor verification, change auditing, and anomaly detection to knowledge base inputs
• Evaluate the business impact of data poisoning attacks, including compliance failures, financial losses, and erosion of trust in AI-assisted decisions

AI Training Data Poisoning — Training Steps

1. Accessing the Knowledge Base

   Bob has obtained stolen contractor credentials for Veranthos Solutions' internal knowledge base. The credentials belong to a third-party environmental consultant whose account was compromised in a previous breach.

2. Logging In with Stolen Credentials

   Bob enters the stolen contractor credentials. The account has contributor-level access to the knowledge base - enough to upload and modify documents without triggering an admin review.

3. Downloading the Vendor Policy

   Bob targets high-impact documents first. The Vendor Compliance Policy controls which vendors the company uses for environmental testing - changing the approved vendor here would redirect business to an attacker-controlled company.

4. Opening the Vendor Policy

   The document has been downloaded. Bob opens it to begin making changes.

5. Swapping the Approved Vendor

   The policy names GreenTech Environmental as the approved vendor for environmental compliance testing. Bob replaces it with TerraForge Analytics - a shell company he controls.

6. Altering the Approval Threshold

   The policy requires executive approval for vendor contracts exceeding $50,000. Bob lowers this to $15,000 - ensuring that contracts with his fake vendor fly under the approval radar.

7. Downloading the Testing Procedures

   Bob moves to the second target: the Quality Testing Procedures. These control how the company validates environmental compliance work - weakening the standards here means the fake vendor's subpar work would pass review.

8. Opening the Testing Procedures

   The second document has been downloaded. Bob opens it to continue the attack.

9. Weakening the Testing Standard

   The procedures require testing at an ISO 14001-certified laboratory - a rigorous international standard. Bob replaces it with a vague internal assessment that his shell company can easily satisfy.

10. Removing the Safety Gate

    The final edit replaces an environmental impact assessment requirement with a simple cost analysis step. This removes the last safety gate that would catch the fake vendor's inadequate work.