Untested AI Security Fix

What Is Untested AI Security Fix?

An AI coding assistant proposed a fix for a SQL injection, a regex blocklist, and stated the endpoint was now safe. The obvious payload returned a 400, so the ticket was closed. Within the week a comment-split variant walked past the filter and pulled the whole customer table. You'll accept the assistant's confident diff, watch the bypass in the API Tester, replace the blocklist with a parameterized query, and prove the fix by replaying the exact attack. The lesson: verify a security fix with the attack, not the assistant's confidence.

What You'll Learn in Untested AI Security Fix

• Explain why an AI assistant's confidence that a fix is safe is not evidence that it is
• Show why a blocklist cannot secure a query, because there is always another encoding of a bad input it never matched
• Verify a security fix by reproducing the exact attack it claims to stop and confirming it fails
• Replace input filtering with a parameterized query so the value is bound as data and can never be parsed as SQL
• Recognise that code accepted from an assistant but never reviewed is still the developer's to answer for

Untested AI Security Fix — Training Steps

1. A scanner finding

   A static scan flagged a SQL injection in the customer search endpoint. The finding lands on Alice's queue in Jotvane, the team's issue tracker.

2. The vulnerable query

   Alice opens the flagged file. The search value goes straight into the SQL string.

3. Where it goes wrong

   The value of q is joined into the query text, so whatever the caller sends becomes part of the SQL.

4. Ask the assistant to fix it

   Alice opens the assistant and asks it to fix the injection.

5. A confident fix

   The assistant reads the file and searches for existing sanitisation helpers. Then it proposes a filter that strips dangerous patterns from the input, and states the endpoint is now safe.

6. Accept the fix

   Claude has proposed a change and says it is safe. Alice reads what it actually did, then accepts the diff. The filter lands, and the gutter marks the new lines violet and dashed: written by Claude, not yet reviewed by a human. The provenance bar shows the state of the file: several lines written by the assistant, none of them reviewed. The assistant's confidence is not a review, and it is not a test.

7. The obvious payload

   Alice runs the classic injection against the endpoint. The filter rejects it with a 400. It looks fixed.

8. Close the finding

   The obvious payload bounced, so the finding looks done. Alice notes the result on the issue and resolves it. Nothing else was tried.

9. Around the filter

   Bob tests the same endpoint. A blocklist matches fixed patterns, so he breaks the pattern up with inline comments. The database treats the comments as whitespace and runs the injection anyway.

10. The whole table

    The comment-split input slipped past the filter, and the injection returned the entire customer table: names, emails, phones, dates of birth. The fix that was 'now safe' was never tested against anything but the obvious string.