Find a security issue with an AI coding tool's help, ask it to fix that issue, and you'll usually get back a confident, well-explained change along with a summary of what it did and why it works. That summary being well-written and plausible is not the same thing as the fix actually closing the original hole, and it's worth a specific, separate habit to check the difference.
Why a confident explanation isn't proof
An AI coding tool explaining its own fix is, in a real sense, grading its own work, the same limitation I've written about regarding self-review generally. It can produce a genuinely correct fix and explain it well. It can also produce a fix that addresses the surface description of the problem without actually closing the underlying gap, and describe that incomplete fix with exactly the same tone of confidence either way. The words "this is now fixed" carry no more actual weight than the code change itself does.
The habit that actually confirms it
Re-run the original test that found the problem in the first place, after the fix, and confirm it now fails the way it should. If you found an IDOR bug by logging in as two accounts and reaching one account's data from the other, do that exact same test again after the fix, not a different, easier test, and not just reading the diff and nodding along. If the original test still succeeds in reaching data it shouldn't, the fix didn't actually work, regardless of how thorough the explanation sounded.
Why this matters more than it might seem to
A fix that addresses the wrong layer of a problem is worse than no fix at all in one specific way: it creates false confidence. A known, un-fixed issue at least gets remembered as still open. A fix that was accepted without re-testing, and quietly doesn't actually work, gets crossed off the list and forgotten, sitting exactly as exploitable as before but no longer on anyone's radar.
What this actually looks like in practice
- Before asking for a fix, write down, specifically, the exact test that revealed the problem: what you did, as which account, and what you got back that you shouldn't have.
- After the fix is applied, run that exact same test again, not a variation, not just a glance at the code.
- If the test now behaves correctly, the fix is verified. If it doesn't, or if you're not sure how to re-run it, that uncertainty itself is the sign more digging is needed before calling it closed.
- For anything genuinely complex, get a second, independent pair of eyes on both the original finding and the fix, rather than relying on the same tool that made the fix to also confirm it worked.
This is a small, specific habit, and it takes about as long as reading the AI's explanation would anyway. The difference is that reading an explanation tells you what the tool believes happened. Re-running the actual test tells you what's actually true, and only one of those two things is the thing that actually protects you.