Hollow AI-Written Tests

What Is Hollow AI-Written Tests?

An AI assistant can write tests that pass without checking the thing that matters. You will accept twelve green tests for an admin-only endpoint, watch an ordinary user read the whole payroll through it, and discover that every test signed in as an admin and one asserted only that a status code exists. You will write the missing negative test, watch it fail against the shipped code, then fix the check. You will leave able to read assistant-written tests and assert the case that must be refused.

What You'll Learn in Hollow AI-Written Tests

• Recognise that passing assistant-written tests prove only the cases that were written, not the ones that were skipped
• Explain why statement coverage counts statements executed, not behaviours checked, so a high number can still miss the branch that leaks
• Read a test to see what identity it signs in as and what it actually asserts, rather than trusting a green result
• Write the negative test for an access-control rule and watch it fail against the broken code before fixing it
• Verify an authorization fix by confirming the refused request returns a 403 and the negative test now passes

Hollow AI-Written Tests — Training Steps

1. Salaries must be admin-only

   Your team lead's ask is at the top of your queue in Jotvane, the team's issue tracker. The new salaries endpoint has to be locked down to admins, and she wants tests before it ships on Friday.

2. Open the endpoint

   The editor loads your new role check, the salaries route and the empty test file. requireRole is the middleware you wrote this morning: a route names the role it needs, and the check is meant to turn everyone else away.

3. Your check, just written

   The salaries route is where your check does its job. Open it to see where requireRole sits.

4. Ask for the tests

   Open Claude and ask it to write the tests for the salaries endpoint, the way your lead asked.

5. Twelve tests, all green

   Claude reads the route and the role check, then writes the test file and reports back: twelve tests, all passing, 94% coverage.

6. Accept the tests

   Accepting lands Claude's test file. The gutter marks every line violet and unreviewed: written by the assistant, not yet read by a human.

7. Green, so it ships

   Run the tests. If they match Claude's report, twelve pass and coverage reads 94%, and the endpoint looks covered enough to ship for Friday.

8. Anyone can read the payroll

   Bob works at Farrowlin with an ordinary account: no admin role, no special access. Curious about the new salaries endpoint, he points the API Tester at it and sends his own session token, the way any signed-in request would.

9. HR noticed

   The HR lead has filed an issue. Someone with no payroll role pulled the full salary list through the API, and she wants to know how the endpoint let them.

10. Read what the tests check

    The provenance bar says who wrote the tests, and that nobody has read them. Read them now: what they sign in as, and what they actually assert. Then mark Claude's lines reviewed.